Since the project's inception, I've missed a clear separation between
default and third-party modules.
This increases complexity within the project (exclude `modules`, but not
`modules/default`), but the mixed use is particularly problematic in
Docker setups.
Therefore, with this pull request, I'm moving the default modules to a
different directory.
~~I've chosen `default/modules`, but I'm not bothered about it;
`defaultmodules` or something similar would work just as well.~~
Changed to `defaultmodules`.
Let me know if there's a majority in favor of this change.
- remove param `--enable-features=UseOzonePlatform` in start electron
tests (as we did already in `package.json`)
- update node versions in github workflows, remove `22.21.1`, add `25.x`
- fix formatting in tests
- update dependencies including electron to v40
This is still a draft PR because most calendar electron tests are not
running which is caused by the electron update from `v39.3.0` to
`v40.0.0`. Maybe @KristjanESPERANTO has an idea ...
---------
Co-authored-by: Kristjan ESPERANTO <35647502+KristjanESPERANTO@users.noreply.github.com>
## Summary
PR [#3976](https://github.com/MagicMirrorOrg/MagicMirror/pull/3976)
introduced smart HTTP error handling for the Calendar module. This PR
extracts that HTTP logic into a central `HTTPFetcher` class.
Calendar is the first module to use it. Follow-up PRs would migrate
Newsfeed and maybe even Weather.
**Before this change:**
- ❌ Each module had to implemented its own `fetch()` calls
- ❌ No centralized retry logic or backoff strategies
- ❌ No timeout handling for hanging requests
- ❌ Error detection relied on fragile string parsing
**What this PR adds:**
- ✅ Unified HTTPFetcher class with intelligent retry strategies
- ✅ Modern AbortController with configurable timeout (default 30s)
- ✅ Proper undici Agent for self-signed certificates
- ✅ Structured error objects with translation keys
- ✅ Calendar module migrated as first consumer
- ✅ Comprehensive unit tests with msw (Mock Service Worker)
## Architecture
**Before - Decentralized HTTP handling:**
```
Calendar Module Newsfeed Module Weather Module
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ fetch() own │ │ fetch() own │ │ fetch() own │
│ retry logic │ │ basic error │ │ no retry │
│ error parse │ │ handling │ │ client-side │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└───────────────────────┴───────────────────────┘
▼
External APIs
```
**After - Centralized with HTTPFetcher:**
```
┌─────────────────────────────────────────────────────┐
│ HTTPFetcher │
│ • Unified retry strategies (401/403, 429, 5xx) │
│ • AbortController timeout (30s) │
│ • Structured errors with translation keys │
│ • undici Agent for self-signed certs │
└────────────┬──────────────┬──────────────┬──────────┘
│ │ │
┌───────▼───────┐ ┌────▼─────┐ ┌──────▼──────┐
│ Calendar │ │ Newsfeed │ │ Weather │
│ ✅ This PR │ │ future │ │ future │
└───────────────┘ └──────────┘ └─────────────┘
│ │ │
└──────────────┴──────────────┘
▼
External APIs
```
## Complexity Considerations
**Does HTTPFetcher add complexity?**
Even if it may look more complex, it actually **reduces overall
complexity**:
- **Calendar already has this logic** (PR #3976) - we're extracting, not
adding
- **Alternative is worse:** Each module implementing own logic = 3× the
code
- **Better testability:** 443 lines of tests once vs. duplicating tests
for each module
- **Standards-based:** Retry-After is RFC 7231, not custom logic
## Future Benefits
**Weather migration (future PR):**
Moving Weather from client-side to server-side will enable:
- **Same robust error handling** - Weather gets 429 rate-limiting, 5xx
backoff for free
- **Simpler architecture** - No proxy layer needed
Moving the weather modules from client-side to server-side will be a big
undertaking, but I think it's a good strategy. Even if we only move the
calendar and newsfeed to the new HTTP fetcher and leave the weather as
it is, this PR still makes sense, I think.
## Breaking Changes
**None**
----
I am eager to hear your opinion on this 🙂
Adapts calendar module to node-ical changes and fixes a bug with moved
full-day recurring events in eastern timezones.
## Changes
### 1. Update node-ical to 0.23.1
- Includes upstream fixes for UNTIL UTC validation errors from CalDAV
servers (reported by @rejas in PR #4010)
- Changes to `getDateKey()` behavior for VALUE=DATE events (now uses
local date components)
- Fixes issue with malformed DURATION values (reported by MagicMirror
user here: https://github.com/jens-maus/node-ical/issues/381)
### 2. Remove dead code
- Removed ineffective UNTIL modification code (rule.options is read-only
in rrule-temporal)
- The code attempted to extend UNTIL for all-day events but had no
effect
### 3. Fix recurrence lookup for full-day events
node-ical changed the behavior of `getDateKey()` - it now uses local
date components for VALUE=DATE events instead of UTC. This broke
recurrence override lookups for full-day events in eastern timezones.
**Why it broke:**
- **before node-ical update:** Both node-ical and MagicMirror used UTC →
keys matched ✅
- **after node-ical update:** node-ical uses local date (RFC 5545
conform), MagicMirror still used UTC → **mismatch** ❌
**Example:**
- Full-day recurring event on October 12 in Europe/Berlin (UTC+2)
- node-ical 0.23.1 stores override with key: `"2024-10-12"` (local date)
- MagicMirror looked for key: `"2024-10-11"` (from UTC: Oct 11 22:00)
- **Result:** Moved event not found, appears on wrong date
**Solution:** Adapt to node-ical's new behavior by using local date
components for full-day events, UTC for timed events.
**Note:** This is different from previous timezone fixes - those
addressed event generation, this fixes the lookup of recurrence
overrides.
## Background
node-ical 0.23.0 switched from `rrule` to `rrule-temporal`, introducing
breaking changes. Version 0.23.1 fixed the UNTIL validation issue and
formalized the `getDateKey()` behavior for DATE vs DATE-TIME values,
following RFC 5545 specification that DATE values represent local
calendar dates without timezone context.
Updating `node-ical` and adapt logic to new behaviour.
## Problem
node-ical 0.23.0 switched from `rrule.js` to `rrule-temporal`, changing
how recurring event dates are returned. Our code assumed the old
behavior where dates needed manual timezone conversion.
## Solution
Updated `getMomentsFromRecurringEvent()` in `calendarfetcherutils.js`:
- Removed `tzid = null` clearing (no longer needed)
- Simplified timed events: `moment.tz(date, eventTimezone)` instead of
`moment.tz(date, "UTC").tz(eventTimezone, true)`
- Kept UTC component extraction for full-day events to prevent date
shifts
Fixes **full-day recurring events showing on wrong day** in timezones
west of UTC (reported in #4003).
**Root cause**: `moment.tz(date, eventTimezone).startOf("day")`
interprets UTC midnight as local time:
- `2025-11-03T00:00:00.000Z` in America/Chicago (UTC-6)
- → Converts to `2025-11-02 18:00:00` (6 hours back)
- → `.startOf("day")` → `2025-11-02 00:00:00` ❌ **Wrong day!**
**Impact**: The bug affects:
- All timezones west of UTC (UTC-1 through UTC-12): Americas, Pacific
- Timezones east of UTC (UTC+1 through UTC+12): Europe, Asia, Africa -
work correctly
- UTC itself - works correctly
The issue was introduced with commit c2ec6fc2 (#3976), which fixed the
time but broke the date. This PR fixes both.
| | Result | Day | Time | Notes |
|----------|--------|-----|------|-------|
| **Before c2ec6fc2** | `2025-11-03 05:00:00 Monday` | ✅ | ❌ | Wrong
time, but correct day |
| **Current (c2ec6fc2)** | `2025-11-02 00:00:00 Sunday` | ❌ (west of
UTC)<br>✅ (east of UTC) | ✅ | Wrong day - visible bug! |
| **This fix** | `2025-11-03 00:00:00 Monday` | ✅ | ✅ | Correct in all
timezones |
Note: While the old logic had incorrect timing, it produced the correct
calendar day due to how it handled UTC offsets. The current logic fixed
the timing issue but introduced the more visible calendar day bug.
### Solution
Extract UTC date components and interpret as local calendar dates:
```javascript
const utcYear = date.getUTCFullYear();
const utcMonth = date.getUTCMonth();
const utcDate = date.getUTCDate();
return moment.tz([utcYear, utcMonth, utcDate], eventTimezone);
```
### Testing
To prevent this from happening again in future refactorings, I wrote a
test for it.
```bash
npm test -- tests/unit/modules/default/calendar/calendar_fetcher_utils_spec.js
```
for issue #3971 add checksum to test if event list changed to
reduce/eliminate no change screen update
fixes#3971
crc32 checksum created in node helper, easy require, vs trying to do in
browser.
added to socket notification payload, used in browser
The bottom line of this PR is, that it fixes an issue and simplifies the
code by dealing with the TODOs in the code.
For review, I suggest looking at each commit individually. If there are
too many changes for a PR, let me know and I'll split it up
🙂
## 1. [fix(calendar): prevent excessive fetching with smart refresh
strategy](8892cd3d5a)
- Add lastFetch timestamp tracking to CalendarFetcher
- Add shouldRefetch() method with configurable minimum interval
(default: 3 minutes)
- When reusing existing fetcher: fetch if data is stale (>3 min),
otherwise broadcast cached events
- Prevents double broadcasts to consuming modules while maintaining
fresh data
- Balances rate limit prevention (Issue
https://github.com/MagicMirrorOrg/MagicMirror/issues/3971) with data
freshness on user reload
- Prevents excessive fetching during rapid reloads (e.g., Fully Kiosk
screensaver use case)
- Allows fresh calendar data when enough time has passed since last
fetch
## 2. [refactor(calendar): simplify event exclusion
logic](d507aba82d)
- Extract filtering logic from `shouldEventBeExcluded` into new helper
`checkEventAgainstFilter`
- Simplify the main loop in `shouldEventBeExcluded
It resolves a TODO from the comments in the code:
* `This seems like an overly complicated way to exclude events based on
the title.`
## 3. [refactor(calendar): extract recurring event expansion
logic](d510160bd2)
This change separates the expansion of recurring events from the main
filtering loop into a new helper function 'expandRecurringEvent'.
It resolves two TODOs from the comments in the code:
- `This should be a separate function`
- `This should create an event per moment so we can change anything we
want`
This improves code readability, reduces complexity in 'filterEvents',
and allows for cleaner handling of individual recurrence instances.
## 4. [refactor(calendar): simplify recurring event
handling](b04f716cc0)
- Simplify 'getMomentsFromRecurringEvent' using modern syntax
- Improve handling of full-day events across different timezones
## 5. [test(calendar): fix UNTIL date in fullday_until.ics
fixture](1d762b2ade)
The issue was with the UNTIL date being May 4th while DTSTART was May
5th. This created an invalid recurrence rule where the end date came
before the start date.
The fix only adjusts the UNTIL date from May 4th to May 5th, so it
matches the start date.
## Changes
- Replace `indexOf()` with `startsWith()` for cleaner protocol detection
- Use `URL` API for robust cache-busting parameter handling
- Add HTTP response validation and improved error logging
- Add JSDoc type annotations for better documentation
- Remove unused `urlSuffix` instance variable
- Add unit tests
- Fix `.gitignore` pattern
## Motivation
After merging #3967, I noticed some potential for improving reliability
and user experience related to the method `loadComplimentFile`. With
these changes the method now validates URLs upfront to catch
configuration errors early, checks HTTP status codes to detect server
issues (404/500), and provides specific error messages that help users
troubleshoot problems.
The complexity of the code does not really increase with the changes. On
the contrary, the method should now be more intuitive to understand.
## Testing
Added unit tests for `loadComplimentFile()` to validate the
improvements:
- HTTP error handling
- Cache-busting
Since E2E tests already cover the happy path, these unit tests focus on
error cases and edge cases.
## Additional Fix
While adding the test file, I discovered that the `.gitignore` pattern
`modules` was incorrectly matching `tests/unit/modules/`, preventing
test files from being tracked. Changed to `/modules` to only match the
root directory.
Checks if `this.config.remoteFile.includes` already contains a `?`
- If it does, uses `&` to append the dummy parameter
- If it doesn't, uses `?` to start a new query string
- Added new pt.json and pt-br.json in alert/translations
- Updated main pt.json (global translations)
- Updated alert.js to load new languages
- Added entry to CHANGELOG.md
---------
Co-authored-by: veeck <gitkraken@veeck.de>
1. Convert CalendarFetcher from legacy constructor function pattern to
ES6 class (which simplifies future migration from CommonJS to ES
modules).
2. Implement targeted HTTP error handling with smart retry strategies
for common calendar feed issues:
- 401/403: Extended retry delay (5× interval, min 30 min)
- 429: Retry-After header parsing with 15 min fallback
- 5xx: Exponential backoff (2^count, max 3 retries)
- 4xx: Extended retry (2× interval, min 15 min)
- Add serverErrorCount tracking for exponential backoff
- Error messages now include specific HTTP status codes and calculated
retry delays for better debugging and user feedback
Previously, CalendarFetcher did not respond appropriately to HTTP
errors, continuing to hammer endpoints without backoff, potentially
overloading servers and triggering rate limits. This refactoring
implements respectful retry strategies that adapt to server responses
and reduce unnecessary load.
Maybe we could later centralize the HTTP error handling and use it for
weather and newsfeed as well.
The PR was inspired by having worked on the calendar fetcher for
MMM-CalendarExt2, where there was already better error handling.
When I saw PR #3951, I wondered why `cspell` didn't catch these typos
before. Unfortunately, the default modules were excluded from the check.
I have corrected this with these changes.
This even revealed a code error in
`modules/default/weather/providers/overrideWrapper.js`:
- before: `fetchEatherHourly`
- after: `fetchWeatherHourly`
## CI Log Suppression
**Two-level approach for clean test output:**
1. **Suppress debug/info logs**: Call `logger.setLogLevel("ERROR")` in
CI to hide verbose logging
2. **Suppress intentional error logs**: Set `mmTestMode` flag and check
it before logging errors that are part of test assertions (e.g., testing
error handling in `git_helper.js` and `server_functions.js`)
This keeps CI output clean and makes genuine failures immediately
visible, while preserving full logging for local development.
**Before:** 1348 log lines with verbose debug/info output
**After:** 168 log clean lines with only test results
## Calendar Symbol Test Stability
Convert the calendar symbol test from external URL (`calendarlabs.com`)
to existing local mock file (`12_events.ics`). This eliminates CI
timeouts caused by external dependencies and improves test reliability.
The test still validates the same symbol array feature but now runs
faster and deterministically without network dependencies.
This is just to reduce a little noise in the dev console. The following
line will not appear with this change:
```shell
module.js:480 Check MagicMirror² version for module 'calendar' - Minimum version: 2.1.0 - Current version: 2.34.0-
```
Since version 2.1.0 is so old, we can surely throw it out without
concern.
The envcanada provider in the default Weather module was fixed in MM
v2.33.0 to use a new URL hierarchy that Environment Canada implemented
to access weather data for Canadian locations. Subsequent to this
provider update, Environment Canada has implemented one further update
to their URL hierarchy to make it easier to access 'current day' weather
data. Tis change was raised in Issue #3912 as a Bug, which is addressed
in this Provider update. There are no Magic Mirror UI changes from this
update.
The fix is to add one additional element to the URL used to access
Environment Canada XML-based weather data.
This PR is also taking the opportunity to make one further small fix to
how windspeed is handled in this Provider. Most of the time, Env Canada
provides an expected numeric value. There are instances, however, where
the value provided is 'calm', which the Weather module does not expect.
The Provider code has been changed to detect a 'calm' windspeed and
convert it to '0' for the purposes of the Weather module. Note that in
the world of weather/climate analysis, a windspeed of 'calm' is used as
a synonym for a windspeed of 0.
Note that a ChangeLog entry is included in this PR.
- removes the external unmaintained `module-alias` dependency ->
reducing complexity and risk
- introduces a small internal alias mechanism for `logger` and
`node_helper`
- preserves backward compatibility for existing 3rd‑party modules
- should simplify a future ESM migration of MagicMirror
I'm confident that it shouldn't cause any problems, but we could also
consider including it in the release after next. What do you think?
This PR is inspired by PR #2934 - so thanks to @thesebas! 🙇😃
Earlier in 2025, Environment Canada changed the process to access
weather data for Canadian cities. This change was raised in Issue #3822
as a Bug, which is addressed in this Provider update. There are no Magic
Mirror UI changes from this update.
The 'old' method to access Environment Canada involved accessing a
static URL based on a City identifier which would result in an XML
document containing the appropriate weather data.
The 'new' method is a 2 step process. The first step is to access a
time-sensitive URL that contains a list of links to various cities that
have weather data available. The second step requires finding the
correct city in that list based on a City identifier, and then accessing
that unique URL to access an XML document containing the appropriate
weather data.
The changes made to the envcanada Provider code are solely aimed at
using the new 2-step method to access a specified City's weather data.
Since the resulting XML document structure has not changed, no other
code in envcanada required changes.
Note that a ChangeLog entry is included in this PR.
---------
Co-authored-by: Kristjan ESPERANTO <35647502+KristjanESPERANTO@users.noreply.github.com>
Co-authored-by: veeck <gitkraken@veeck.de>
The current logic never showed any different temperature than the
current one, so I looked into a more "explained" formula and found one
in the HomeAssistant forums.
Fixes#3253
Adds a configuration option to overwrite the default `User-Agent` header
that is send at least by the calendar and news module. Allows other
modules to use the individual user agent as well.
The configuration accepts either a string or a function:
```
var config =
{
...
userAgent: 'Mozilla/5.0 (My User Agent)',
...
}
```
or
```
var config =
{
...
userAgent: () => 'Mozilla/5.0 (My User Agent)',
...
}
```
---------
Co-authored-by: Veeck <github@veeck.de>
Co-authored-by: veeck <gitkraken@veeck.de>
Co-authored-by: Karsten Hassel <hassel@gmx.de>
Co-authored-by: Kristjan ESPERANTO <35647502+KristjanESPERANTO@users.noreply.github.com>
Just some normal maintainance after the holidays
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: veeck <gitkraken@veeck.de>
The weather e2e tests are failing sometimes, failing is not really
reproducable.
After changing `updateDom(0)` to `updateDom(300)` in `weather.js` it
seems to work (we will se if it really works in the long term).
This PR contains some other weather e2e changes/cleanups/simplifying.
to avoid exception in eslint config.
Background: If someone has a copy of the `modules` folder in his setup
eslint fails because the file `debug.js` uses `console` statements. I
need the `modules` folder renamed in my docker setup so this test always
fails. I think this is a simple solution which has no impact.
1. Base your pull requests against the `develop` branch.
Done
2. Include these infos in the description:
- Does the pull request solve a **related** issue?
No
- If so, can you reference the issue like this `Fixes #<issue_number>`?
N/A
- What does the pull request accomplish? Use a list if needed.
With some combinations of sunrise and sunset times (usually when the
time till rise/set is >9:59), the information will break across multiple
lines. This prevents that by adding CSS.
- If it includes major visual changes please add screenshots.
I don't consider it major.
3. Please run `npm run lint:prettier` before submitting so that style
issues are fixed.
Done
4. Don't forget to add an entry about your changes to the CHANGELOG.md
file.
Done
---------
Co-authored-by: veeck <gitkraken@veeck.de>
- fix newsfeed and calendar e2e tests so they don't need `--forceExit`
anymore
- `--forceExit` should stay in test runs to avoid running until test
limit in case of errors
- remove mocking console, not needed anymore
- configFactory-stuff should not run in browser (otherwise we get errors
`[ReferenceError: exports is not defined]`)
