Forecast arrival of pilgrims in Santiago

[JavierLeon]

Hi all,

I have access to data on arrivals of pilgrims to Santiago by day, country, age, gender, departure point, route, reason and means used. Based on this data, I am analysing it to develop a machine learning model that will allow me to predict the arrival of pilgrims to Santiago.

Which of the dimensions do you think most influence the prediction? Right now I am choosing day, total pilgrims and top10 countries/pilgrims. In addition, I am adding features such as day of the week, week, month, festivities, holy week, week of the virgin of August, week of the constitution or Christmas. I am also taking into account the Año Santo and COVID years.

Do you think that the weather forecast influences the choice of season? In my case, this has never been the case.

As soon as I have the first results, I will publish them here.

Thank you and Buen Camino!

 

[Vacajoe]

Fascinating! I don’t think weather has any effect. I think the biggest factor is when pilgrims start walking (weekends, holidays, religious observances) but you won’t be able to judge that with your data. However, I think whatever you can figure out will be fascinating

 

[Bradypus]

Fascinating! I don’t think weather has any effect.

Probably a very limited effect. Those who are travelling long distances for their Caminos are probably committed well in advance of weather forecasts. And last month's record Compostela figures happened despite some atrocious weather!

 

[CalgaryLynn]

Hi all,

I have access to data on arrivals of pilgrims to Santiago by day, country, age, gender, departure point, route, reason and means used. Based on this data, I am analysing it to develop a machine learning model that will allow me to predict the arrival of pilgrims to Santiago.

Which of the dimensions do you think most influence the prediction? Right now I am choosing day, total pilgrims and top10 countries/pilgrims. In addition, I am adding features such as day of the week, week, month, festivities, holy week, week of the virgin of August, week of the constitution or Christmas. I am also taking into account the Año Santo and COVID years.

Do you think that the weather forecast influences the choice of season? In my case, this has never been the case.

As soon as I have the first results, I will publish them here.

Thank you and Buen Camino!

I know for myself, weather was the first thing I looked at when deciding when to go on the Camino. I tried to pick the least crowded time with the best weather predictions and luckily for me, it worked. I had 11 days of beautiful weather. Luck was on my side.

 

[Eckart.S]

You could use the weather of the preceeding week, as most people start from Sarria and walk for less than a week.

 

[JavierLeon]

Fascinating! I don’t think weather has any effect. I think the biggest factor is when pilgrims start walking (weekends, holidays, religious observances) but you won’t be able to judge that with your data. However, I think whatever you can figure out will be fascinating

We have to make a lot of assumptions, for example, that pilgrims begin and end, that is, not all those who do it by sections are considered. But if something you say is true, there is an imbalance to finish from Thursday to Saturday I understand that to enjoy Santiago on the weekend,

That’s the point to include country and Camino by day to find the influence between the variables… let see the results

 

[JavierLeon]

Probably a very limited effect. Those who are travelling long distances for their Caminos are probably committed well in advance of weather forecasts. And last month's record Compostela figures happened despite some atrocious weather!

That’s been alway my case: If I have the time, I don't care about the weather. On my first way, a thousand years ago, a Japanese couple in theirs 80 taught me this way. It was raining and no one dared to go out. They put on the poncho, greeted me in the Japanese way and told me:
- it's dawned a perfect day to walk
- but if it's raining...
- yes, but there's no other... so it's perfect.

Let see the results…

 

[JavierLeon]

You could use the weather of the preceeding week, as most people start from Sarria and walk for less than a week.

You’re right. I will try to link the place of departure (short, medium, long distance) with the mean days of the Camino and their historical weather... many assumptions ;-)

Let see the results… thanks!

 

[mgnswaus]

I know it’s an impossible task but I’ve always been curious to know how many Caminos are walked and not registered by collecting a Compostela. I suspect many ‘addicts’ like me don’t collect Compostelas. I also suspect the number would be significant on top of what the recorded data suggest….

 

[Vacajoe]

A bit off-topic, but it’s about data: is there ever a “one day count” where all the pilgrims at albergues on a specific day are counted? While it would miss those at hotels, etc, it would be fascinating.

 

[Sminn]

I would probably group the countries by continent as well to see if there was a significance. You may find certain countries/continents travel at certain times of the year. I agree that the weather wouldn’t play much of a part but holidays may. It will be interesting to see what the data says.

 

[JavierLeon]

I would probably group the countries by continent as well to see if there was a significance. You may find certain countries/continents travel at certain times of the year. I agree that the weather wouldn’t play much of a part but holidays may. It will be interesting to see what the data says.

I’ll include that aggregation by geographical areas to see if there is any impact. Thank you!

 

[JavierLeon]

A bit off-topic, but it’s about data: is there ever a “one day count” where all the pilgrims at albergues on a specific day are counted? While it would miss those at hotels, etc, it would be fascinating.

I wish that were possible because it would be a fundamental source of data to allocate/promote actions by areas but I'm afraid it's very complicated to add it on the same record...

 

[Rita Flower]

Probably a very limited effect. Those who are travelling long distances for their Caminos are probably committed well in advance of weather forecasts. And last month's record Compostela figures happened despite some atrocious weather!

Pilgrims from overseas will come not matter what the weather - we have bought out tickets and we will come. However, when I walked the Portuguese in 2018 the weather was cold and wet for a long time and we were told the numbers were down, mainly European pilgrims who find it easier to change plans.

 

[WalkingInGers]

Hi all,

I have access to data on arrivals of pilgrims to Santiago by day, country, age, gender, departure point, route, reason and means used. Based on this data, I am analysing it to develop a machine learning model that will allow me to predict the arrival of pilgrims to Santiago.

Which of the dimensions do you think most influence the prediction? Right now I am choosing day, total pilgrims and top10 countries/pilgrims. In addition, I am adding features such as day of the week, week, month, festivities, holy week, week of the virgin of August, week of the constitution or Christmas. I am also taking into account the Año Santo and COVID years.

Do you think that the weather forecast influences the choice of season? In my case, this has never been the case.

As soon as I have the first results, I will publish them here.

Thank you and Buen Camino!

For many of us who live in Europe, the weather is certainly a factor in terms of when to do a camino. True, two weeks of vacation here and there does help but seriously...would most of us leave to walk a camino under 40C or under extremely adverse weather conditions?

 

[Rick of Rick and Peg]

You have one dependent variable, arrivals in Santiago, and many independent variables that affect the dependent variable such as time of year. From this you want to find a mathematical formula where the input varables yield a result. Look to regession analysis.

Click fo Spanish Wikipedia article
Click for English Wikipedia article

 

[JavierLeon]

You have one dependent variable, arrivals in Santiago, and many independent variables that affect the dependent variable such as time of year. From this you want to find a mathematical formula where the input varables yield a result. Look to regession analysis.

Click fo Spanish Wikipedia article
Click for English Wikipedia article

Thank you Rick. I’m using time series with ML/DL to capture season patterns and trends.

 

[kateonthecamino]

Really interested to see your results!

Since day of week starting seems to be a big factor anecdotally (particularly weekends, holidays, 1st of the month) it might be worth making some assumptions to generate an estimated start date based off starting point and day of arrival. Obviously it would be very inaccurate on an individual level, but the numbers should be big enough to see trends (though it would assume one continuous camino, which could be more/less true for some nationalities than others )

 

[t2andreo]

There is technically feasible way to do this accurately. It requires installing active signal antennas at each of the entries into Santiago from the many Camino routes ending at Santiago.

I like the idea of solar-powered sensors installed at key mojones (distance markers) out on Camino routes, or at key streets at the established entries into Santiago.

As a pilgrim passes by these “screens,” the unique ESN (Electronic Serial Number) can be read and tallied, by portal entry point. Each number would only be allowed to be counted once in a defined period.

So someone remaining at Santiago would not be counted multiple times.

What is tallied is “X” number of unique ESNs that enter Santiago from each Camino portal. It’s not perfect. But it is a start.

You could place similar reader at the “choke points” on each route several days before Santiago. For example, at the exits from: Sarria, Tui, etc. these choke points are places where pilgrims will start from.

Just thinking out loud -so to speak.

 

[David Tallan]

There is technically feasible way to do this accurately. It requires installing active signal antennas at each of the entries into Santiago from the many Camino routes ending at Santiago.

I like the idea of solar-powered sensors installed at key mojones (distance markers) out on Camino routes, or at key streets at the established entries into Santiago.

As a pilgrim passes by these “screens,” the unique ESN (Electronic Serial Number) can be read and tallied, by portal entry point. Each number would only be allowed to be counted once in a defined period.

So someone remaining at Santiago would not be counted multiple times.

What is tallied is “X” number of unique ESNs that enter Santiago from each Camino portal. It’s not perfect. But it is a start.

You could place similar reader at the “choke points” on each route several days before Santiago. For example, at the exits from: Sarria, Tui, etc. these choke points are places where pilgrims will start from.

Just thinking out loud -so to speak.

You could put little RFID chips in credenciales that could be read.

 

[Wandershot]

A good predictor could be albergue vacancy rate in key starting towns (St Jean, Astorga, Sarria). This could be a good indicator of +X days to arrival in SdC. The Wise Pilgrim app has just added a voluntary feature where albs can self-report.

 

[JavierLeon]

Really interested to see your results!

Since day of week starting seems to be a big factor anecdotally (particularly weekends, holidays, 1st of the month) it might be worth making some assumptions to generate an estimated start date based off starting point and day of arrival. Obviously it would be very inaccurate on an individual level, but the numbers should be big enough to see trends (though it would assume one continuous camino, which could be more/less true for some nationalities than others )

Thank you Kate. Good point. I'm not really clear right now whether the day I want to finish or the day I want to start would have more influence. I think the first one... but I'm including it in the analysis.
Buen Camino!

 

[Kailua920]

Hi all,

I have access to data on arrivals of pilgrims to Santiago by day, country, age, gender, departure point, route, reason and means used. Based on this data, I am analysing it to develop a machine learning model that will allow me to predict the arrival of pilgrims to Santiago.

Which of the dimensions do you think most influence the prediction? Right now I am choosing day, total pilgrims and top10 countries/pilgrims. In addition, I am adding features such as day of the week, week, month, festivities, holy week, week of the virgin of August, week of the constitution or Christmas. I am also taking into account the Año Santo and COVID years.

Do you think that the weather forecast influences the choice of season? In my case, this has never been the case.

As soon as I have the first results, I will publish them here.

Thank you and Buen Camino!

Is it possible to collect the average expected kilometers per day per pilgrim as a data input?

If not, if you have access to older data that includes route start, age, mode, and finish date you could apply the same assumption parameters in generating the predicted arrival output (but you likely thought of that).

There may be a way to combine 3rd party data (you’d need to obtain) as an overlay set to generate assumptions about the speed of pilgrims through each Camino. Sample data could be from gronze bed availability stats, or something similar.

I would assume that Spain or villages along each route do not use video monitoring or surveillance along the Caminos (thankfully!!), however, perhaps Burgos, Leon and other major cities on each route do. Video with object recognition (people counting with backpacks) would be a data point.

Just answering this question, makes me realize how much I need to do a Camino pilgrimage!!

Good luck…sounds like an interesting and valuable project! Katrina

 

[t2andreo]

You could put little RFID chips in credenciales that could be read.

That would work too. But there are too many credencial versions and issuing authorities I think.

The beauty of using the cellular phone ESN is that ALL cellular phones: smart, old school, flip or soap bar style, use the same technology to “ping” cellular towers. The ESN is one of the standard pieces of unique information that is always read by a cellular tower to initiate a “handshake.”

Even if you have a non-compatible SIM, the tower will read the ESN to determine if it can handle the call request or not. As long as the device is powered on, the ESN will be interrogated. If that is done, the ability to count is supported.

Just developing the thought.

 

[David Tallan]

That would work too. But there are too many credencial versions and issuing authorities I think.

The beauty of using the cellular phone ESN is that ALL cellular phones: smart, old school, flip or soap bar style, use the same technology to “ping” cellular towers. The ESN is one of the standard pieces of unique information that is always read by a cellular tower to initiate a “handshake.”

Even if you have a non-compatible SIM, the tower will read the ESN to determine if it can handle the call request or not. As long as the device is powered on, the ESN will be interrogated. If that is done, the ability to count is supported.

Just developing the thought.

My thought was that the advantage of the credencial was that all pilgrims carry them and there is a central authority (the cathedral) that exerts some control over them already. Even if different groups issue them, they do so according to rules set by the cathedral. Getting all pilgrims to install something in their phones that would identify them as pilgrims by the ESN may be more of a challenge.

 

[t2andreo]

Nothing needs to be installed on any cellular phone. The ability to interrogate the built-in circuitry and the embedded ESN is integral to all cellular phones around the globe.

Conversely, the credencial RFID chip has limitations. Not all pilgrims heading to Santiago have one. We assume this is the case, but I think not. A number of pilgrims are just out for the experience. The credencial is only needed if you want to stay at an albergue. They not required for commercial lodging.

Several years ago, I floated the idea of affixing a tamper-proof silicone waist band, with the RFID chip at designated locations where one would start their Camino. That chip could be read passively as a pilgrim walked by a sensor..

I have now realized that there are as many logistical problems with the tamper-proof wristband, as there are with the RFID chip credencial. I no longer support the tamper-proof wrist band.

I still believe that simply leveraging the technology that is ubiquitous and reliable is closer to the optimal solution.

If folks want to rely on other, more “squishy” population estimates, that’s fine by me. I am just trying to move the ball down the pitch…

In any event, knowing how many pilgrims are coming is a net good thing.

Hope this helps the dialogue,

Tom

 

[David Tallan]

Nothing needs to be installed on any cellular phone. The ability to interrogate the built-in circuitry and the embedded ESN is integral to all cellular phones around the globe.

Conversely, the credencial RFID chip has limitations. Not all pilgrims heading to Santiago have one. We assume this is the case, but I think not. A number of pilgrims are just out for the experience. The credencial is only needed if you want to stay at an albergue. They not required for commercial lodging.

Several years ago, I floated the idea of affixing a tamper-proof silicone waist band, with the RFID chip at designated locations where one would start their Camino. That chip could be read passively as a pilgrim walked by a sensor..

I have now realized that there are as many logistical problems with the tamper-proof wristband, as there are with the RFID chip credencial. I no longer support the tamper-proof wrist band.

I still believe that simply leveraging the technology that is ubiquitous and reliable is closer to the optimal solution.

If folks want to rely on other, more “squishy” population estimates, that’s fine by me. I am just trying to move the ball down the pitch…

In any event, knowing how many pilgrims are coming is a net good thing.

Hope this helps the dialogue,

Tom

But how would the interrogation distinguish pilgrims from non-pilgrims? I'm probably missing something obvious here.

 

[JavierLeon]

Nothing needs to be installed on any cellular phone. The ability to interrogate the built-in circuitry and the embedded ESN is integral to all cellular phones around the globe.

Conversely, the credencial RFID chip has limitations. Not all pilgrims heading to Santiago have one. We assume this is the case, but I think not. A number of pilgrims are just out for the experience. The credencial is only needed if you want to stay at an albergue. They not required for commercial lodging.

Several years ago, I floated the idea of affixing a tamper-proof silicone waist band, with the RFID chip at designated locations where one would start their Camino. That chip could be read passively as a pilgrim walked by a sensor..

I have now realized that there are as many logistical problems with the tamper-proof wristband, as there are with the RFID chip credencial. I no longer support the tamper-proof wrist band.

I still believe that simply leveraging the technology that is ubiquitous and reliable is closer to the optimal solution.

If folks want to rely on other, more “squishy” population estimates, that’s fine by me. I am just trying to move the ball down the pitch…

In any event, knowing how many pilgrims are coming is a net good thing.

Hope this helps the dialogue,

Tom

Thank you for your idea... but I am afraid it is completely out of the scope of this exercise.

There are so many variables and lack of a complete vision of the Camino as a whole that it is very difficult to propose this type of solution. To begin with, talking to the telephone companies to access data from the antennas is simply impossible... and we would have to know who is a pilgrim and who is not so a central register of "pilgrims en route" would also be needed with all the privacy issues that would arise.

But the Camino is also fragmented in Regions and even in provinces that barely share data beyond some strategic plan paid for by Europe that never comes to anything.

And if we focus on the register of overnight stays associated with the Compostela, things do not improve, as there is no central register in which data is aggregated. In many cases, the registers are still on paper....

In addition, as has already been mentioned in the thread, the casuistry of pilgrims is infinite: those who do it complete, those who do stages, those who stay in public hostels, those who stay in private, those who stay in the homes of acquaintances and friends, ... it's a mess. Almost the only thing that identifies them ends up being the Compostela.

It is an exciting topic but I'm afraid that I have long since given up on having more or less curated and coherent data. Some time ago, a collaborator of the hostels of Saint Jean told me that for some time they had been identifying that their numbers of pilgrims who started there did not coincide with those who arrived, even including the jumps of years...

Whether we like it or not, the most complete source of data is the Pilgrim's Office at the Cathedral and it also does not include those who finish and do not ask for the certificate (like me sometimes).

Maybe estimations can also be made of the average duration of a Camino to look for weather data on those dates to see if there is an influence but for now I will keep it simple to see if the prediction models make sense or not.

So far, I'm working with this features with date = day

RangeIndex: 7809 entries, 0 to 7808 (days)
total_pilgrims_daily
top_camino_XX_camino, top_pilgrims_XX_camino (01 to 10)
top_origin_XX_origin, top_pilgrims_XX_origin (01 to 10)
top_country_XX_country, top_pilgrims_XX_country (01 to 10)
top_age_XX, top_pilgrims_age_XX (01 to 04)
day_of_week, week_of_year, month, year
weekly_cumulative_pilgrims, monthly_cumulative_pilgrims, yearly_cumulative_pilgrims
ma_1_days, ma_3_days, ma_5_days, ma_7_days, ma_14_days, ma_21_days (mean average)
Is_Holiday, is_easter, is_christmas, is_virgen_august_weeks, is_constitution_weeks, is_jacobean_years, is_COVID

Let see the results... for now I have it automated to collect the data and add the features as I have time for analysis.

Buen Camino... and thanks again!

 

[JavierLeon]

Really interested to see your results!

Since day of week starting seems to be a big factor anecdotally (particularly weekends, holidays, 1st of the month) it might be worth making some assumptions to generate an estimated start date based off starting point and day of arrival. Obviously it would be very inaccurate on an individual level, but the numbers should be big enough to see trends (though it would assume one continuous camino, which could be more/less true for some nationalities than others )

Hi Kate,

There is a very important bias in the days of the week that most pilgrims arrive: Friday > Saturday > Thursday > Sunday... it seems that pilgrims want to make sure they enjoy Santiago ;-)

Buen Camino!!