Errors on handling data received lead to request limit exceeded

[jstammi]

For Hyundai 2020-Kona, EU/Germany, using modified version of monitor R4.10.0.
Running the monitor infinitely (interval: 30 minutes, max force sync 15) within a docker container.

Whenever the local handling of the data having received results in an error, at least in infinite running mode, this leads to an as-fast-as-possible-retry loop including repeated login to the servers and fetching vehicles (status). Until the request limit is exceeded.
Afterwards the nominal error handling steps in with (in my case) trying 2 times to reach out for the servers (original impl: 15x), with 1 minute sleeping in between, until giving up and sleeping for the configured time until running again.

In short this is what happened to me (this time with having mis-configured the monitor wrt mqtt), everything for today, 27.10.2025:

1. 01:01:40 start of monitor, checking for changes every 30 minutes
2. 06:01:43 1st change was detected.
3. The monitor tries to send the change to mqtt and fails, immediately re-tries this with the need to login, detects the same change again, fails again to send to mqtt, ... (each roundtrip ~7-10 seconds)
4. ... this repeats in total until 06:17:01. Then an ODO change is detected, resulting in modified loop with additionally triggering a forced sync before "writing the new csv" (in fact it seems to1st send to mqtt and fail there, as the csv file does never change). Each roundtrip now takes ~30s
5. Until 07:02:41, then the allowed number of requests was exceeded
6. this stops the loop before and starts a low frequent new one: wait one minute, then try again with same outcome (with original implementation this would repeat another 13 times), then stop and record the iteration as a failed one: error count: 1
7. then the next 2 iterations, each time after ~30 minutes waiting, until 08:02:41, the vehicle is checked without identifying a change, nor failing
8. repeat steps 6, but only once (08:24:37 until 08:25:42)
9. since then monitor checks again for changes without detecting any, nor failing again (08:54:37 until 16:24:38, on-going)

Some weeks ago already similar was triggered as reported by #84, different cause, same symptoms wrt requests exceeded that day.

I tried hard, but yet I still understand all of the variants of behaviour of this. But for sure there is a critical problem on proper error handling
local errors while handling the data must not trigger for fetching the data again, nor lead to request-limit-exceeded
(this issue is NOT about solving the local data handling problem - last time file encoding, this time mqtt, next time maybe e.g. filesystem full - different stories)

Here are 2 logs (private data redacted) that I captured:

1. the logging (output to console) of monitor.py. This shows one of my modifications: a DEBUG output on each iteration within the loop inside handle_vehicles, showing the login parameter and current retries: bug89-monitor.log
2. A trace of most http(s) requests by the api impl towards hyundai and openstreetmap servers (I have available also the POST bodies and response texts if some detail is needed additionally, but no headers) - this is another modification that I implemented for other reasons, but that I consider of possibly being of help here, too. This trace misses the _get_cookies requests as they use a dedicated session, that I miss to copy the nominal session's hooks to, yet: bug89-requests.log

[ZuinigeRijder]

Ok, when using infinite I start a subprocess for the commands, e.g. summary and dailystats. Although those are exited when paho_mqtt is not installed, monitor.py keeps continuing in run_commands(). Also when paho_mqtt is not installed, at least in that situation, also monitor.py should exit. I will see if I can make a solution for this situation.

Whenever there is an error in the subprocess commands, what would you expect that monitor.py does? Also in case of other errors?

[jstammi]

The problem is not the concrete cause (mqtt here). The problem is the exception handling in general: any exception, independent if it is a monitor-internal one or raised while fetching data from external, is handled in the same place. This must be devided up as handling differs.

Problems raised on calling VehicleManager methods are sensitive wrt request limit and one need to be very carefull with retries. I am not yet sure about it and unfortunately the requests trace is not fully complete (when running into requests exceeded, there were still some requests not captured as they still used a transient requests.Session, that I did not hook in, yet). But if the requests limit of 200 is correct and I count requests in the trace within the relevant time interval, then it seems as if any request to (for EU Kona) https://prd.eu-ccapi.hyundai.com:8080/api/v1/spa/vehicles and below is counting at least. Please correct me if I'm wrong with that or if there is more in depth knowledge already about the way this limit reduces by calling what web api endpoints.
The call to check_and_refresh_token followed by update_all_vehicles_with_cached_state results in at least 5 web api reqeusts (token still valid, no change in data detected).

So for exception raised there, it is mandatory to keep with a low frequent retry strategy.

And such retries must not be triggered by exceptions raised by locally handling the data afterwards. But at the moment it is. This is a general problem at the moment in the current monitor code. And up to now the code is still too new to me to be able to propose an implementation solution there. I just can provide with the reommendation to handle "local" exception separate from those raised within VehicleManager. With the goal that no exception raised in the monitor data handling part must cause any retry for fetching data using the VehicleManager.

For the concrete question:

• if the application is of no use at all, there is a fundamental error in it, then it should exit (at the moment this is prevented from by the catch-all exception handling afais)
• if just one component is causing a problem, e.g. publishing to mqtt, but writing csv files is working, then I would not expect an exiting of the application but error logging and keep going. Btw including still writing the csv files, which is currently not the case with the mqtt problem. Imho the error handling isolation is missing there, too. But the error handling on that level could be improved also later, this is less critical I guess.

[ZuinigeRijder]

But if the requests limit of 200 is correct and I count requests in the trace within the relevant time interval, then it seems as if any request to (for EU Kona) https://prd.eu-ccapi.hyundai.com:8080/api/v1/spa/vehicles and below is counting at least. Please correct me if I'm wrong with that or if there is more in depth knowledge already about the way this limit reduces by calling what web api endpoints. The call to check_and_refresh_token followed by update_all_vehicles_with_cached_state results in at least 5 web api reqeusts (token still valid, no change in data detected).

I run monitor.py infinite every 15 minutes and I do generally not have a problem with the request limit. So probably for Europe the limit is NOT 200 and subsequent API calls are not counted individually too.

I did some better error handling in R4.13.0, but of course, if something is really wrong, it could be that the request limit is eventually reached.