Merge pull request #10 from celebrateyang/sync-api-from-upstream

Sync api from upstream

api/README.md

@@ -11,12 +11,9 @@ we recommend [deploying your own instance](/docs/run-an-instance.md) if you wish
 
 you can read [the api documentation here](/docs/api.md).
 
-> [!WARNING]
-> the v7 public api (/api/json) will be shut down on **november 11th, 2024**.
-> you can access documentation for it [here](https://github.com/imputnet/cobalt/blob/7/docs/api.md).
-
 ## supported services
-this list is not final and keeps expanding over time. if support for a service you want is missing, create an issue (or a pull request 👀).
+this list is not final and keeps expanding over time!
+if the desired service isn't supported yet, feel free to create an appropriate issue (or a pull request 👀).
 
 | service           | video + audio | only audio | only video | metadata | rich file names |
 | :--------         | :-----------: | :--------: | :--------: | :------: | :-------------: |
@@ -39,12 +36,13 @@ this list is not final and keeps expanding over time. if support for a service y
 | twitter/x         | ✅            | ✅         | ✅         | ➖         | ➖              |
 | vimeo             | ✅            | ✅         | ✅         | ✅         | ✅              |
 | vk videos & clips | ✅            | ❌         | ✅         | ✅         | ✅              |
+| xiaohongshu       | ✅            | ✅         | ✅         | ➖         | ➖              |
 | youtube           | ✅            | ✅         | ✅         | ✅         | ✅              |
 
 | emoji   | meaning                 |
 | :-----: | :---------------------- |
 | ✅      | supported               |
-| ➖      | impossible/unreasonable |
+| ➖      | unreasonable/impossible |
 | ❌      | not supported           |
 
 ### additional notes or features (per service)
@@ -71,38 +69,35 @@ as long as you:
 - provide a link to the license and indicate if changes to the code were made, and
 - release the code under the **same license**
 
-## acknowledgements
+## open source acknowledgements
 ### ffmpeg
-cobalt heavily relies on ffmpeg for converting and merging media files. it's an absolutely amazing piece of software offered for anyone for free, yet doesn't receive as much credit as it should.
+cobalt relies on ffmpeg for muxing and encoding media files. ffmpeg is absolutely spectacular and we're privileged to have the ability to use it for free, just like anyone else. we believe it should be way more recognized.
 
 you can [support ffmpeg here](https://ffmpeg.org/donations.html)!
 
-#### ffmpeg-static
-we use [ffmpeg-static](https://github.com/eugeneware/ffmpeg-static) to get binaries for ffmpeg depending on the platform.
-
-you can support the developer via various methods listed on their github page! (linked above)
-
 ### youtube.js
-cobalt relies on [youtube.js](https://github.com/LuanRT/YouTube.js) for interacting with the innertube api, it wouldn't have been possible without it.
+cobalt relies on **[youtube.js](https://github.com/LuanRT/YouTube.js)** for interacting with youtube's innertube api, it wouldn't have been possible without this package.
 
-you can support the developer via various methods listed on their github page! (linked above)
+you can support the developer via various methods listed on their github page!
+(linked above)
 
 ### many others
-cobalt also depends on:
+cobalt-api also depends on:
 
-- [content-disposition-header](https://www.npmjs.com/package/content-disposition-header) to simplify the provision of `content-disposition` headers.
+- **[content-disposition-header](https://www.npmjs.com/package/content-disposition-header)** to simplify the provision of `content-disposition` headers.
-- [cors](https://www.npmjs.com/package/cors) to manage cross-origin resource sharing within expressjs.
+- **[cors](https://www.npmjs.com/package/cors)** to manage cross-origin resource sharing within expressjs.
-- [dotenv](https://www.npmjs.com/package/dotenv) to load environment variables from the `.env` file.
+- **[dotenv](https://www.npmjs.com/package/dotenv)** to load environment variables from the `.env` file.
-- [esbuild](https://www.npmjs.com/package/esbuild) to minify the frontend files.
+- **[express](https://www.npmjs.com/package/express)** as the backbone of cobalt servers.
-- [express](https://www.npmjs.com/package/express) as the backbone of cobalt servers.
+- **[express-rate-limit](https://www.npmjs.com/package/express-rate-limit)** to rate limit api endpoints.
-- [express-rate-limit](https://www.npmjs.com/package/express-rate-limit) to rate limit api endpoints.
+- **[ffmpeg-static](https://www.npmjs.com/package/ffmpeg-static)** to get binaries for ffmpeg depending on the platform.
-- [hls-parser](https://www.npmjs.com/package/hls-parser) to parse `m3u8` playlists for certain services.
+- **[hls-parser](https://www.npmjs.com/package/hls-parser)** to parse HLS playlists according to spec (very impressive stuff).
-- [ipaddr.js](https://www.npmjs.com/package/ipaddr.js) to parse ip addresses (for rate limiting).
+- **[ipaddr.js](https://www.npmjs.com/package/ipaddr.js)** to parse ip addresses (used for rate limiting).
-- [nanoid](https://www.npmjs.com/package/nanoid) to generate unique (temporary) identifiers for each requested stream.
+- **[nanoid](https://www.npmjs.com/package/nanoid)** to generate unique identifiers for each requested tunnel.
-- [node-cache](https://www.npmjs.com/package/node-cache) to cache stream info in server ram for a limited amount of time.
+- **[set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser)** to parse cookies that cobalt receives from certain services.
-- [psl](https://www.npmjs.com/package/psl) as the domain name parser.
+- **[undici](https://www.npmjs.com/package/undici)** for making http requests.
-- [set-cookie-parser](https://www.npmjs.com/package/set-cookie-parser) to parse cookies that cobalt receives from certain services.
+- **[url-pattern](https://www.npmjs.com/package/url-pattern)** to match provided links with supported patterns.
-- [undici](https://www.npmjs.com/package/undici) for making http requests.
+- **[zod](https://www.npmjs.com/package/zod)** to lock down the api request schema.
-- [url-pattern](https://www.npmjs.com/package/url-pattern) to match provided links with supported patterns.
+- **[@datastructures-js/priority-queue](https://www.npmjs.com/package/@datastructures-js/priority-queue)** for sorting stream caches for future clean up (without redis).
+- **[@imput/psl](https://www.npmjs.com/package/@imput/psl)** as the domain name parser, our fork of [psl](https://www.npmjs.com/package/psl).
 
 ...and many other packages that these packages rely on.

api/package.json

@@ -1,7 +1,7 @@
 {
     "name": "@imput/cobalt-api",
     "description": "save what you love",
-    "version": "10.4.3",
+    "version": "11.0.2",
     "author": "imput",
     "exports": "./src/cobalt.js",
     "type": "module",
@@ -11,7 +11,6 @@
     "scripts": {
         "start": "node src/cobalt",
         "test": "node src/util/test",
-        "token:youtube": "node src/util/generate-youtube-tokens",
         "token:jwt": "node src/util/generate-jwt-secret"
     },
     "repository": {
@@ -30,18 +29,17 @@
         "content-disposition-header": "0.6.0",
         "cors": "^2.8.5",
         "dotenv": "^16.0.1",
-        "esbuild": "^0.14.51",
+        "express": "^4.21.2",
-        "express": "^4.21.1",
         "express-rate-limit": "^7.4.1",
         "ffmpeg-static": "^5.1.0",
         "hls-parser": "^0.10.7",
         "ipaddr.js": "2.2.0",
-        "nanoid": "^4.0.2",
+        "mime": "^4.0.4",
-        "node-cache": "^5.1.2",
+        "nanoid": "^5.0.9",
         "set-cookie-parser": "2.6.0",
         "undici": "^5.19.1",
         "url-pattern": "1.0.3",
-        "youtubei.js": "^12.2.0",
+        "youtubei.js": "^13.4.0",
         "zod": "^3.23.8"
     },
     "optionalDependencies": {

api/src/config.js

@@ -1,77 +1,29 @@
 import { getVersion } from "@imput/version-info";
-import { services } from "./processing/service-config.js";
+import { loadEnvs, validateEnvs, setupEnvWatcher } from "./core/env.js";
-import { supportsReusePort } from "./misc/cluster.js";
 
 const version = await getVersion();
 
-const disabledServices = process.env.DISABLED_SERVICES?.split(',') || [];
+const env = loadEnvs();
-const enabledServices = new Set(Object.keys(services).filter(e => {
-    if (!disabledServices.includes(e)) {
-        return e;
-    }
-}));
-
-const env = {
-    apiURL: process.env.API_URL || '',
-    apiPort: process.env.API_PORT || 9000,
-    tunnelPort: process.env.API_PORT || 9000,
-
-    listenAddress: process.env.API_LISTEN_ADDRESS,
-    freebindCIDR: process.platform === 'linux' && process.env.FREEBIND_CIDR,
-
-    corsWildcard: process.env.CORS_WILDCARD !== '0',
-    corsURL: process.env.CORS_URL,
-
-    cookiePath: process.env.COOKIE_PATH,
-
-    rateLimitWindow: (process.env.RATELIMIT_WINDOW && parseInt(process.env.RATELIMIT_WINDOW)) || 60,
-    rateLimitMax: (process.env.RATELIMIT_MAX && parseInt(process.env.RATELIMIT_MAX)) || 20,
-
-    durationLimit: (process.env.DURATION_LIMIT && parseInt(process.env.DURATION_LIMIT)) || 10800,
-    streamLifespan: (process.env.TUNNEL_LIFESPAN && parseInt(process.env.TUNNEL_LIFESPAN)) || 90,
-
-    processingPriority: process.platform !== 'win32'
-        && process.env.PROCESSING_PRIORITY
-        && parseInt(process.env.PROCESSING_PRIORITY),
-
-    externalProxy: process.env.API_EXTERNAL_PROXY,
-
-    turnstileSitekey: process.env.TURNSTILE_SITEKEY,
-    turnstileSecret: process.env.TURNSTILE_SECRET,
-    jwtSecret: process.env.JWT_SECRET,
-    jwtLifetime: process.env.JWT_EXPIRY || 120,
-
-    sessionEnabled: process.env.TURNSTILE_SITEKEY
-                        && process.env.TURNSTILE_SECRET
-                        && process.env.JWT_SECRET,
-
-    apiKeyURL: process.env.API_KEY_URL && new URL(process.env.API_KEY_URL),
-    authRequired: process.env.API_AUTH_REQUIRED === '1',
-    redisURL: process.env.API_REDIS_URL,
-    instanceCount: (process.env.API_INSTANCE_COUNT && parseInt(process.env.API_INSTANCE_COUNT)) || 1,
-    keyReloadInterval: 900,
-
-    enabledServices,
-}
 
 const genericUserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36";
 const cobaltUserAgent = `cobalt/${version} (+https://github.com/imputnet/cobalt)`;
 
+export const canonicalEnv = Object.freeze(structuredClone(process.env));
 export const setTunnelPort = (port) => env.tunnelPort = port;
 export const isCluster = env.instanceCount > 1;
+export const updateEnv = (newEnv) => {
+    // tunnelPort is special and needs to get carried over here
+    newEnv.tunnelPort = env.tunnelPort;
 
-if (env.sessionEnabled && env.jwtSecret.length < 16) {
+    for (const key in env) {
-    throw new Error("JWT_SECRET env is too short (must be at least 16 characters long)");
+        env[key] = newEnv[key];
+    }
 }
 
-if (env.instanceCount > 1 && !env.redisURL) {
+await validateEnvs(env);
-    throw new Error("API_REDIS_URL is required when API_INSTANCE_COUNT is >= 2");
+
-} else if (env.instanceCount > 1 && !await supportsReusePort()) {
+if (env.envFile) {
-    console.error('API_INSTANCE_COUNT is not supported in your environment. to use this env, your node.js');
+    setupEnvWatcher();
-    console.error('version must be >= 23.1.0, and you must be running a recent enough version of linux');
-    console.error('(or other OS that supports it). for more info, see `reusePort` option on');
-    console.error('https://nodejs.org/api/net.html#serverlistenoptions-callback');
-    throw new Error('SO_REUSEPORT is not supported');
 }
 
 export {

api/src/core/api.js

@@ -8,18 +8,21 @@ import jwt from "../security/jwt.js";
 import stream from "../stream/stream.js";
 import match from "../processing/match.js";
 
-import { env, isCluster, setTunnelPort } from "../config.js";
+import { env } from "../config.js";
 import { extract } from "../processing/url.js";
-import { Green, Bright, Cyan } from "../misc/console-text.js";
+import { Bright, Cyan } from "../misc/console-text.js";
 import { hashHmac } from "../security/secrets.js";
 import { createStore } from "../store/redis-ratelimit.js";
 import { randomizeCiphers } from "../misc/randomize-ciphers.js";
 import { verifyTurnstileToken } from "../security/turnstile.js";
 import { friendlyServiceName } from "../processing/service-alias.js";
-import { verifyStream, getInternalStream } from "../stream/manage.js";
+import { verifyStream } from "../stream/manage.js";
 import { createResponse, normalizeRequest, getIP } from "../processing/request.js";
+import { setupTunnelHandler } from "./itunnel.js";
+
 import * as APIKeys from "../security/api-keys.js";
 import * as Cookies from "../processing/cookie/manager.js";
+import * as YouTubeSession from "../processing/helpers/youtube-session.js";
 
 const git = {
     branch: await getBranch(),
@@ -45,35 +48,38 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
     const startTime = new Date();
     const startTimestamp = startTime.getTime();
 
-    const serverInfo = JSON.stringify({
+    const getServerInfo = () => {
+        return JSON.stringify({
             cobalt: {
                 version: version,
                 url: env.apiURL,
                 startTime: `${startTimestamp}`,
-            durationLimit: env.durationLimit,
                 turnstileSitekey: env.sessionEnabled ? env.turnstileSitekey : undefined,
                 services: [...env.enabledServices].map(e => {
                     return friendlyServiceName(e);
                 }),
             },
             git,
-    })
+        });
+    }
+
+    const serverInfo = getServerInfo();
 
     const handleRateExceeded = (_, res) => {
-        const { status, body } = createResponse("error", {
+        const { body } = createResponse("error", {
             code: "error.api.rate_exceeded",
             context: {
                 limit: env.rateLimitWindow
             }
         });
-        return res.status(status).json(body);
+        return res.status(429).json(body);
     };
 
     const keyGenerator = (req) => hashHmac(getIP(req), 'rate').toString('base64url');
 
     const sessionLimiter = rateLimit({
-        windowMs: 60000,
+        windowMs: env.sessionRateLimitWindow * 1000,
-        limit: 10,
+        limit: env.sessionRateLimit,
         standardHeaders: 'draft-6',
         legacyHeaders: false,
         keyGenerator,
@@ -89,19 +95,19 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
         keyGenerator: req => req.rateLimitKey || keyGenerator(req),
         store: await createStore('api'),
         handler: handleRateExceeded
-    })
+    });
 
     const apiTunnelLimiter = rateLimit({
-        windowMs: env.rateLimitWindow * 1000,
+        windowMs: env.tunnelRateLimitWindow * 1000,
-        limit: (req) => req.rateLimitMax || env.rateLimitMax,
+        limit: env.tunnelRateLimitMax,
         standardHeaders: 'draft-6',
         legacyHeaders: false,
-        keyGenerator: req => req.rateLimitKey || keyGenerator(req),
+        keyGenerator: req => keyGenerator(req),
         store: await createStore('tunnel'),
         handler: (_, res) => {
-            return res.sendStatus(429)
+            return res.sendStatus(429);
         }
-    })
+    });
 
     app.set('trust proxy', ['loopback', 'uniquelocal']);
 
@@ -173,11 +179,12 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
                 return fail(res, "error.api.auth.jwt.invalid");
             }
 
-            if (!jwt.verify(token)) {
+            if (!jwt.verify(token, getIP(req, 32))) {
                 return fail(res, "error.api.auth.jwt.invalid");
             }
 
             req.rateLimitKey = hashHmac(token, 'rate');
+            req.isSession = true;
         } catch {
             return fail(res, "error.api.generic");
         }
@@ -219,7 +226,7 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
         }
 
         try {
-            res.json(jwt.generate());
+            res.json(jwt.generate(getIP(req, 32)));
         } catch {
             return fail(res, "error.api.generic");
         }
@@ -242,6 +249,7 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
         if (!parsed) {
             return fail(res, "error.api.link.invalid");
         }
+
         if ("error" in parsed) {
             let context;
             if (parsed?.context) {
@@ -255,13 +263,23 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
                 host: parsed.host,
                 patternMatch: parsed.patternMatch,
                 params: normalizedRequest,
+                isSession: req.isSession ?? false,
             });
 
             res.status(result.status).json(result.body);
         } catch {
             fail(res, "error.api.generic");
         }
-    })
+    });
+
+    app.use('/tunnel', cors({
+        methods: ['GET'],
+        exposedHeaders: [
+            'Estimated-Content-Length',
+            'Content-Disposition'
+        ],
+        ...corsConfig,
+    }));
 
     app.get('/tunnel', apiTunnelLimiter, async (req, res) => {
         const id = String(req.query.id);
@@ -292,35 +310,11 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
         }
 
         return stream(res, streamInfo);
-    })
+    });
-
-    const itunnelHandler = (req, res) => {
-        if (!req.ip.endsWith('127.0.0.1')) {
-            return res.sendStatus(403);
-        }
-
-        if (String(req.query.id).length !== 21) {
-            return res.sendStatus(400);
-        }
-
-        const streamInfo = getInternalStream(req.query.id);
-        if (!streamInfo) {
-            return res.sendStatus(404);
-        }
-
-        streamInfo.headers = new Map([
-            ...(streamInfo.headers || []),
-            ...Object.entries(req.headers)
-        ]);
-
-        return stream(res, { type: 'internal', ...streamInfo });
-    };
-
-    app.get('/itunnel', itunnelHandler);
 
     app.get('/', (_, res) => {
         res.type('json');
-        res.status(200).send(serverInfo);
+        res.status(200).send(env.envFile ? getServerInfo() : serverInfo);
     })
 
     app.get('/favicon.ico', (req, res) => {
@@ -340,10 +334,6 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
     setInterval(randomizeCiphers, 1000 * 60 * 30); // shuffle ciphers every 30 minutes
 
     if (env.externalProxy) {
-        if (env.freebindCIDR) {
-            throw new Error('Freebind is not available when external proxy is enabled')
-        }
-
         setGlobalDispatcher(new ProxyAgent(env.externalProxy))
     }
 
@@ -354,7 +344,7 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
     }, () => {
         if (isPrimary) {
             console.log(`\n` +
-                Bright(Cyan("cobalt ")) + Bright("API ^ω⁠^") + "\n" +
+                Bright(Cyan("cobalt ")) + Bright("API ^ω^") + "\n" +
 
                 "~~~~~~\n" +
                 Bright("version: ") + version + "\n" +
@@ -376,19 +366,11 @@ export const runAPI = async (express, app, __dirname, isPrimary = true) => {
         if (env.cookiePath) {
             Cookies.setup(env.cookiePath);
         }
+
+        if (env.ytSessionServer) {
+            YouTubeSession.setup();
+        }
     });
 
-    if (isCluster) {
+    setupTunnelHandler();
-        const istreamer = express();
-        istreamer.get('/itunnel', itunnelHandler);
-        const server = istreamer.listen({
-            port: 0,
-            host: '127.0.0.1',
-            exclusive: true
-        }, () => {
-            const { port } = server.address();
-            console.log(`${Green('[✓]')} cobalt sub-instance running on 127.0.0.1:${port}`);
-            setTunnelPort(port);
-        });
-    }
 }

api/src/core/env.js

@@ -0,0 +1,191 @@
+import { Constants } from "youtubei.js";
+import { services } from "../processing/service-config.js";
+import { updateEnv, canonicalEnv, env as currentEnv } from "../config.js";
+
+import { FileWatcher } from "../misc/file-watcher.js";
+import { isURL } from "../misc/utils.js";
+import * as cluster from "../misc/cluster.js";
+import { Yellow } from "../misc/console-text.js";
+
+const forceLocalProcessingOptions = ["never", "session", "always"];
+
+export const loadEnvs = (env = process.env) => {
+    const disabledServices = env.DISABLED_SERVICES?.split(',') || [];
+    const enabledServices = new Set(Object.keys(services).filter(e => {
+        if (!disabledServices.includes(e)) {
+            return e;
+        }
+    }));
+
+    return {
+        apiURL: env.API_URL || '',
+        apiPort: env.API_PORT || 9000,
+        tunnelPort: env.API_PORT || 9000,
+
+        listenAddress: env.API_LISTEN_ADDRESS,
+        freebindCIDR: process.platform === 'linux' && env.FREEBIND_CIDR,
+
+        corsWildcard: env.CORS_WILDCARD !== '0',
+        corsURL: env.CORS_URL,
+
+        cookiePath: env.COOKIE_PATH,
+
+        rateLimitWindow: (env.RATELIMIT_WINDOW && parseInt(env.RATELIMIT_WINDOW)) || 60,
+        rateLimitMax: (env.RATELIMIT_MAX && parseInt(env.RATELIMIT_MAX)) || 20,
+
+        tunnelRateLimitWindow: (env.TUNNEL_RATELIMIT_WINDOW && parseInt(env.TUNNEL_RATELIMIT_WINDOW)) || 60,
+        tunnelRateLimitMax: (env.TUNNEL_RATELIMIT_MAX && parseInt(env.TUNNEL_RATELIMIT_MAX)) || 40,
+
+        sessionRateLimitWindow: (env.SESSION_RATELIMIT_WINDOW && parseInt(env.SESSION_RATELIMIT_WINDOW)) || 60,
+        sessionRateLimit: (env.SESSION_RATELIMIT && parseInt(env.SESSION_RATELIMIT)) || 10,
+
+        durationLimit: (env.DURATION_LIMIT && parseInt(env.DURATION_LIMIT)) || 10800,
+        streamLifespan: (env.TUNNEL_LIFESPAN && parseInt(env.TUNNEL_LIFESPAN)) || 90,
+
+        processingPriority: process.platform !== 'win32'
+            && env.PROCESSING_PRIORITY
+            && parseInt(env.PROCESSING_PRIORITY),
+
+        externalProxy: env.API_EXTERNAL_PROXY,
+
+        turnstileSitekey: env.TURNSTILE_SITEKEY,
+        turnstileSecret: env.TURNSTILE_SECRET,
+        jwtSecret: env.JWT_SECRET,
+        jwtLifetime: env.JWT_EXPIRY || 120,
+
+        sessionEnabled: env.TURNSTILE_SITEKEY
+                            && env.TURNSTILE_SECRET
+                            && env.JWT_SECRET,
+
+        apiKeyURL: env.API_KEY_URL && new URL(env.API_KEY_URL),
+        authRequired: env.API_AUTH_REQUIRED === '1',
+        redisURL: env.API_REDIS_URL,
+        instanceCount: (env.API_INSTANCE_COUNT && parseInt(env.API_INSTANCE_COUNT)) || 1,
+        keyReloadInterval: 900,
+
+        enabledServices,
+
+        customInnertubeClient: env.CUSTOM_INNERTUBE_CLIENT,
+        ytSessionServer: env.YOUTUBE_SESSION_SERVER,
+        ytSessionReloadInterval: 300,
+        ytSessionInnertubeClient: env.YOUTUBE_SESSION_INNERTUBE_CLIENT,
+        ytAllowBetterAudio: env.YOUTUBE_ALLOW_BETTER_AUDIO !== "0",
+
+        // "never" | "session" | "always"
+        forceLocalProcessing: env.FORCE_LOCAL_PROCESSING ?? "never",
+
+        envFile: env.API_ENV_FILE,
+        envRemoteReloadInterval: 300,
+    };
+}
+
+export const validateEnvs = async (env) => {
+    if (env.sessionEnabled && env.jwtSecret.length < 16) {
+        throw new Error("JWT_SECRET env is too short (must be at least 16 characters long)");
+    }
+
+    if (env.instanceCount > 1 && !env.redisURL) {
+        throw new Error("API_REDIS_URL is required when API_INSTANCE_COUNT is >= 2");
+    } else if (env.instanceCount > 1 && !await cluster.supportsReusePort()) {
+        console.error('API_INSTANCE_COUNT is not supported in your environment. to use this env, your node.js');
+        console.error('version must be >= 23.1.0, and you must be running a recent enough version of linux');
+        console.error('(or other OS that supports it). for more info, see `reusePort` option on');
+        console.error('https://nodejs.org/api/net.html#serverlistenoptions-callback');
+        throw new Error('SO_REUSEPORT is not supported');
+    }
+
+    if (env.customInnertubeClient && !Constants.SUPPORTED_CLIENTS.includes(env.customInnertubeClient)) {
+        console.error("CUSTOM_INNERTUBE_CLIENT is invalid. Provided client is not supported.");
+        console.error(`Supported clients are: ${Constants.SUPPORTED_CLIENTS.join(', ')}\n`);
+        throw new Error("Invalid CUSTOM_INNERTUBE_CLIENT");
+    }
+
+    if (env.forceLocalProcessing && !forceLocalProcessingOptions.includes(env.forceLocalProcessing)) {
+        console.error("FORCE_LOCAL_PROCESSING is invalid.");
+        console.error(`Supported options are are: ${forceLocalProcessingOptions.join(', ')}\n`);
+        throw new Error("Invalid FORCE_LOCAL_PROCESSING");
+    }
+
+    if (env.externalProxy && env.freebindCIDR) {
+        throw new Error('freebind is not available when external proxy is enabled')
+    }
+}
+
+const reloadEnvs = async (contents) => {
+    const newEnvs = {};
+    const resolvedContents = await contents;
+
+    for (let line of resolvedContents.split('\n')) {
+        line = line.trim();
+        if (line === '') {
+            continue;
+        }
+
+        let [ key, value ] = line.split(/=(.+)?/);
+        if (key) {
+            if (value.match(/^['"]/) && value.match(/['"]$/)) {
+                value = JSON.parse(value);
+            }
+
+            newEnvs[key] = value || '';
+        }
+    }
+
+    const candidate = {
+        ...canonicalEnv,
+        ...newEnvs,
+    };
+
+    const parsed = loadEnvs(candidate);
+    await validateEnvs(parsed);
+    updateEnv(parsed);
+    cluster.broadcast({ env_update: resolvedContents });
+}
+
+const wrapReload = (contents) => {
+    reloadEnvs(contents)
+    .catch((e) => {
+        console.error(`${Yellow('[!]')} Failed reloading environment variables at ${new Date().toISOString()}.`);
+        console.error('Error:', e);
+    });
+}
+
+let watcher;
+const setupWatcherFromFile = (path) => {
+    const load = () => wrapReload(watcher.read());
+
+    if (isURL(path)) {
+        watcher = FileWatcher.fromFileProtocol(path);
+    } else {
+        watcher = new FileWatcher({ path });
+    }
+
+    watcher.on('file-updated', load);
+    load();
+}
+
+const setupWatcherFromFetch = (url) => {
+    const load = () => wrapReload(fetch(url).then(r => r.text()));
+    setInterval(load, currentEnv.envRemoteReloadInterval);
+    load();
+}
+
+export const setupEnvWatcher = () => {
+    if (cluster.isPrimary) {
+        const envFile = currentEnv.envFile;
+        const isFile = !isURL(envFile)
+                       || new URL(envFile).protocol === 'file:';
+
+        if (isFile) {
+            setupWatcherFromFile(envFile);
+        } else {
+            setupWatcherFromFetch(envFile);
+        }
+    } else if (cluster.isWorker) {
+        process.on('message', (message) => {
+            if ('env_update' in message) {
+                reloadEnvs(message.env_update);
+            }
+        });
+    }
+}

api/src/core/itunnel.js

@@ -0,0 +1,61 @@
+import stream from "../stream/stream.js";
+import { getInternalTunnel } from "../stream/manage.js";
+import { setTunnelPort } from "../config.js";
+import { Green } from "../misc/console-text.js";
+import express from "express";
+
+const validateTunnel = (req, res) => {
+    if (!req.ip.endsWith('127.0.0.1')) {
+        res.sendStatus(403);
+        return;
+    }
+
+    if (String(req.query.id).length !== 21) {
+        res.sendStatus(400);
+        return;
+    }
+
+    const streamInfo = getInternalTunnel(req.query.id);
+    if (!streamInfo) {
+        res.sendStatus(404);
+        return;
+    }
+
+    return streamInfo;
+}
+
+const streamTunnel = (req, res) => {
+    const streamInfo = validateTunnel(req, res);
+    if (!streamInfo) {
+        return;
+    }
+
+    streamInfo.headers = new Map([
+        ...(streamInfo.headers || []),
+        ...Object.entries(req.headers)
+    ]);
+
+    return stream(res, { type: 'internal', data: streamInfo });
+}
+
+export const setupTunnelHandler = () => {
+    const tunnelHandler = express();
+
+    tunnelHandler.get('/itunnel', streamTunnel);
+
+    // fallback
+    tunnelHandler.use((_, res) => res.sendStatus(400));
+    // error handler
+    tunnelHandler.use((_, __, res, ____) => res.socket.end());
+
+
+    const server = tunnelHandler.listen({
+        port: 0,
+        host: '127.0.0.1',
+        exclusive: true
+    }, () => {
+        const { port } = server.address();
+        console.log(`${Green('[✓]')} internal tunnel handler running on 127.0.0.1:${port}`);
+        setTunnelPort(port);
+    });
+}

api/src/misc/file-watcher.js

@@ -0,0 +1,43 @@
+import { EventEmitter } from 'node:events';
+import * as fs from 'node:fs/promises';
+
+export class FileWatcher extends EventEmitter {
+    #path;
+    #hasWatcher = false;
+    #lastChange = new Date().getTime();
+
+    constructor({ path, ...rest }) {
+        super(rest);
+        this.#path = path;
+    }
+
+    async #setupWatcher() {
+        if (this.#hasWatcher)
+            return;
+
+        this.#hasWatcher = true;
+        const watcher = fs.watch(this.#path);
+        for await (const _ of watcher) {
+            if (new Date() - this.#lastChange > 50) {
+                this.emit('file-updated');
+                this.#lastChange = new Date().getTime();
+            }
+        }
+    }
+
+    read() {
+        this.#setupWatcher();
+        return fs.readFile(this.#path, 'utf8');
+    }
+
+    static fromFileProtocol(url_) {
+        const url = new URL(url_);
+        if (url.protocol !== 'file:') {
+            return;
+        }
+
+        const pathname = url.pathname === '/' ? '' : url.pathname;
+        const file_path = decodeURIComponent(url.host + pathname);
+        return new this({ path: file_path });
+    }
+}

api/src/misc/run-test.js

@@ -23,6 +23,15 @@ export async function runTest(url, params, expect) {
     if (expect.status !== result.body.status) {
         const detail = `${expect.status} (expected) != ${result.body.status} (actual)`;
         error.push(`status mismatch: ${detail}`);
+
+        if (result.body.status === 'error') {
+            error.push(`error code: ${result.body?.error?.code}`);
+        }
+    }
+
+    if (expect.errorCode && expect.errorCode !== result.body?.error?.code) {
+        const detail = `${expect.errorCode} (expected) != ${result.body.error.code} (actual)`
+        error.push(`error mismatch: ${detail}`);
     }
 
     if (expect.code !== result.status) {

api/src/misc/utils.js

@@ -1,8 +1,27 @@
-export function getRedirectingURL(url) {
+import { request } from 'undici';
-    return fetch(url, { redirect: 'manual' }).then((r) => {
+const redirectStatuses = new Set([301, 302, 303, 307, 308]);
-        if ([301, 302, 303].includes(r.status) && r.headers.has('location'))
+
-            return r.headers.get('location');
+export async function getRedirectingURL(url, dispatcher, headers) {
+    const params = {
+        dispatcher,
+        method: 'HEAD',
+        headers,
+        redirect: 'manual'
+    };
+
+    let location = await request(url, params).then(r => {
+        if (redirectStatuses.has(r.statusCode) && r.headers['location']) {
+            return r.headers['location'];
+        }
     }).catch(() => null);
+
+    location ??= await fetch(url, params).then(r => {
+        if (redirectStatuses.has(r.status) && r.headers.has('location')) {
+            return r.headers.get('location');
+        }
+    }).catch(() => null);
+
+    return location;
 }
 
 export function merge(a, b) {
@@ -29,3 +48,16 @@ export function splitFilenameExtension(filename) {
         return [ parts.join('.'), ext ]
     }
 }
+
+export function zip(a, b) {
+    return a.map((value, i) => [ value, b[i] ]);
+}
+
+export function isURL(input) {
+    try {
+        new URL(input);
+        return true;
+    } catch {
+        return false;
+    }
+}

api/src/processing/cookie/manager.js

@@ -12,7 +12,7 @@ const VALID_SERVICES = new Set([
     'instagram_bearer',
     'reddit',
     'twitter',
-    'youtube_oauth'
+    'youtube',
 ]);
 
 const invalidCookies = {};

api/src/processing/create-filename.js

@@ -1,10 +1,25 @@
-const illegalCharacters = ['}', '{', '%', '>', '<', '^', ';', ':', '`', '$', '"', "@", '=', '?', '|', '*'];
+// characters that are disallowed on windows:
+// https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file#naming-conventions
+const characterMap = {
+    '<':  '＜',
+    '>':  '＞',
+    ':':  '：',
+    '"':  '＂',
+    '/':  '／',
+    '\\': '＼',
+    '|': '｜',
+    '?': '？',
+    '*': '＊'
+};
 
-const sanitizeString = (string) => {
+export const sanitizeString = (string) => {
-    for (const i in illegalCharacters) {
+    // remove any potential control characters the string might contain
-        string = string.replaceAll("/", "_").replaceAll("\\", "_")
+    string = string.replace(/[\u0000-\u001F\u007F-\u009F]/g, "");
-                       .replaceAll(illegalCharacters[i], '')
+
+    for (const [ char, replacement ] of Object.entries(characterMap)) {
+        string = string.replaceAll(char, replacement);
     }
+
     return string;
 }
 

api/src/processing/helpers/youtube-session.js

@@ -0,0 +1,81 @@
+import * as cluster from "../../misc/cluster.js";
+
+import { Agent } from "undici";
+import { env } from "../../config.js";
+import { Green, Yellow } from "../../misc/console-text.js";
+
+const defaultAgent = new Agent();
+
+let session;
+
+const validateSession = (sessionResponse) => {
+    if (!sessionResponse.potoken) {
+        throw "no poToken in session response";
+    }
+
+    if (!sessionResponse.visitor_data) {
+        throw "no visitor_data in session response";
+    }
+
+    if (!sessionResponse.updated) {
+        throw "no last update timestamp in session response";
+    }
+
+    // https://github.com/iv-org/youtube-trusted-session-generator/blob/c2dfe3f/potoken_generator/main.py#L25
+    if (sessionResponse.potoken.length < 160) {
+        console.error(`${Yellow('[!]')} poToken is too short and might not work (${new Date().toISOString()})`);
+    }
+}
+
+const updateSession = (newSession) => {
+    session = newSession;
+}
+
+const loadSession = async () => {
+    const sessionServerUrl = new URL(env.ytSessionServer);
+    sessionServerUrl.pathname = "/token";
+
+    const newSession = await fetch(
+        sessionServerUrl,
+        { dispatcher: defaultAgent }
+    ).then(a => a.json());
+
+    validateSession(newSession);
+
+    if (!session || session.updated < newSession?.updated) {
+        cluster.broadcast({ youtube_session: newSession });
+        updateSession(newSession);
+    }
+}
+
+const wrapLoad = (initial = false) => {
+    loadSession()
+    .then(() => {
+        if (initial) {
+            console.log(`${Green('[✓]')} poToken & visitor_data loaded successfully!`);
+        }
+    })
+    .catch((e) => {
+        console.error(`${Yellow('[!]')} Failed loading poToken & visitor_data at ${new Date().toISOString()}.`);
+        console.error('Error:', e);
+    })
+}
+
+export const getYouTubeSession = () => {
+    return session;
+}
+
+export const setup = () => {
+    if (cluster.isPrimary) {
+        wrapLoad(true);
+        if (env.ytSessionReloadInterval > 0) {
+            setInterval(wrapLoad, env.ytSessionReloadInterval * 1000);
+        }
+    } else if (cluster.isWorker) {
+        process.on('message', (message) => {
+            if ('youtube_session' in message) {
+                updateSession(message.youtube_session);
+            }
+        });
+    }
+}

api/src/processing/match-action.js

@@ -5,7 +5,22 @@ import { audioIgnore } from "./service-config.js";
 import { createStream } from "../stream/manage.js";
 import { splitFilenameExtension } from "../misc/utils.js";
 
-export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disableMetadata, filenameStyle, twitterGif, requestIP, audioBitrate, alwaysProxy }) {
+const extraProcessingTypes = ["merge", "remux", "mute", "audio", "gif"];
+
+export default function({
+    r,
+    host,
+    audioFormat,
+    isAudioOnly,
+    isAudioMuted,
+    disableMetadata,
+    filenameStyle,
+    convertGif,
+    requestIP,
+    audioBitrate,
+    alwaysProxy,
+    localProcessing
+}) {
     let action,
         responseType = "tunnel",
         defaultParams = {
@@ -15,13 +30,14 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
             filename: r.filenameAttributes ?
                     createFilename(r.filenameAttributes, filenameStyle, isAudioOnly, isAudioMuted) : r.filename,
             fileMetadata: !disableMetadata ? r.fileMetadata : false,
-            requestIP
+            requestIP,
+            originalRequest: r.originalRequest
         },
         params = {};
 
     if (r.isPhoto) action = "photo";
     else if (r.picker) action = "picker"
-    else if (r.isGif && twitterGif) action = "gif";
+    else if (r.isGif && convertGif) action = "gif";
     else if (isAudioOnly) action = "audio";
     else if (isAudioMuted) action = "muteVideo";
     else if (r.isHLS) action = "hls";
@@ -47,7 +63,7 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
             });
 
         case "photo":
-            responseType = "redirect";
+            params = { type: "proxy" };
             break;
 
         case "gif":
@@ -68,7 +84,8 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
             }
             params = {
                 type: muteType,
-                url: Array.isArray(r.urls) ? r.urls[0] : r.urls
+                url: Array.isArray(r.urls) ? r.urls[0] : r.urls,
+                isHLS: r.isHLS
             }
             if (host === "reddit" && r.typeId === "redirect") {
                 responseType = "redirect";
@@ -82,6 +99,7 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
                 case "twitter":
                 case "snapchat":
                 case "bsky":
+                case "xiaohongshu":
                     params = { picker: r.picker };
                     break;
 
@@ -101,6 +119,7 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
                             filename: `${r.audioFilename}.${audioFormat}`,
                             isAudioOnly: true,
                             audioFormat,
+                            audioBitrate
                         })
                     }
                     break;
@@ -141,6 +160,7 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
                 case "ok":
                 case "vk":
                 case "tiktok":
+                case "xiaohongshu":
                     params = { type: "proxy" };
                     break;
 
@@ -211,5 +231,14 @@ export default function({ r, host, audioFormat, isAudioOnly, isAudioMuted, disab
         params.type = "proxy";
     }
 
-    return createResponse(responseType, {...defaultParams, ...params})
+    // TODO: add support for HLS
+    // (very painful)
+    if (localProcessing && !params.isHLS && extraProcessingTypes.includes(params.type)) {
+        responseType = "local-processing";
+    }
+
+    return createResponse(
+        responseType,
+        { ...defaultParams, ...params }
+    );
 }

api/src/processing/match.js

@@ -28,10 +28,11 @@ import snapchat from "./services/snapchat.js";
 import loom from "./services/loom.js";
 import facebook from "./services/facebook.js";
 import bluesky from "./services/bluesky.js";
+import xiaohongshu from "./services/xiaohongshu.js";
 
 let freebind;
 
-export default async function({ host, patternMatch, params }) {
+export default async function({ host, patternMatch, params, isSession }) {
     const { url } = params;
     assert(url instanceof URL);
     let dispatcher, requestIP;
@@ -69,7 +70,7 @@ export default async function({ host, patternMatch, params }) {
                 r = await twitter({
                     id: patternMatch.id,
                     index: patternMatch.index - 1,
-                    toGif: !!params.twitterGif,
+                    toGif: !!params.convertGif,
                     alwaysProxy: params.alwaysProxy,
                     dispatcher
                 });
@@ -108,10 +109,14 @@ export default async function({ host, patternMatch, params }) {
                 }
 
                 if (url.hostname === "music.youtube.com" || isAudioOnly) {
-                    fetchInfo.quality = "max";
+                    fetchInfo.quality = "1080";
                     fetchInfo.format = "vp9";
                     fetchInfo.isAudioOnly = true;
                     fetchInfo.isAudioMuted = false;
+
+                    if (env.ytAllowBetterAudio && params.youtubeBetterAudio) {
+                        fetchInfo.quality = "max";
+                    }
                 }
 
                 r = await youtube(fetchInfo);
@@ -119,9 +124,8 @@ export default async function({ host, patternMatch, params }) {
 
             case "reddit":
                 r = await reddit({
-                    sub: patternMatch.sub,
+                    ...patternMatch,
-                    id: patternMatch.id,
+                    dispatcher,
-                    user: patternMatch.user
                 });
                 break;
 
@@ -131,7 +135,7 @@ export default async function({ host, patternMatch, params }) {
                     shortLink: patternMatch.shortLink,
                     fullAudio: params.tiktokFullAudio,
                     isAudioOnly,
-                    h265: params.tiktokH265,
+                    h265: params.allowH265,
                     alwaysProxy: params.alwaysProxy,
                 });
                 break;
@@ -157,12 +161,8 @@ export default async function({ host, patternMatch, params }) {
                 isAudioOnly = true;
                 isAudioMuted = false;
                 r = await soundcloud({
-                    url,
+                    ...patternMatch,
-                    author: patternMatch.author,
-                    song: patternMatch.song,
                     format: params.audioFormat,
-                    shortLink: patternMatch.shortLink || false,
-                    accessKey: patternMatch.accessKey || false
                 });
                 break;
 
@@ -227,7 +227,8 @@ export default async function({ host, patternMatch, params }) {
 
             case "facebook":
                 r = await facebook({
-                    ...patternMatch
+                    ...patternMatch,
+                    dispatcher
                 });
                 break;
 
@@ -239,6 +240,15 @@ export default async function({ host, patternMatch, params }) {
                 });
                 break;
 
+            case "xiaohongshu":
+                r = await xiaohongshu({
+                    ...patternMatch,
+                    h265: params.allowH265,
+                    isAudioOnly,
+                    dispatcher,
+                });
+                break;
+
             default:
                 return createResponse("error", {
                     code: "error.api.service.unsupported"
@@ -261,7 +271,7 @@ export default async function({ host, patternMatch, params }) {
             switch(r.error) {
                 case "content.too_long":
                     context = {
-                        limit: env.durationLimit / 60,
+                        limit: parseFloat((env.durationLimit / 60).toFixed(2)),
                     }
                     break;
 
@@ -282,6 +292,14 @@ export default async function({ host, patternMatch, params }) {
             })
         }
 
+        let localProcessing = params.localProcessing;
+
+        const lpEnv = env.forceLocalProcessing;
+
+        if (lpEnv === "always" || (lpEnv === "session" && isSession)) {
+            localProcessing = true;
+        }
+
         return matchAction({
             r,
             host,
@@ -290,10 +308,11 @@ export default async function({ host, patternMatch, params }) {
             isAudioMuted,
             disableMetadata: params.disableMetadata,
             filenameStyle: params.filenameStyle,
-            twitterGif: params.twitterGif,
+            convertGif: params.convertGif,
             requestIP,
             audioBitrate: params.audioBitrate,
             alwaysProxy: params.alwaysProxy,
+            localProcessing,
         })
     } catch {
         return createResponse("error", {

api/src/processing/request.js

@@ -1,7 +1,8 @@
+import mime from "mime";
 import ipaddr from "ipaddr.js";
 
-import { createStream } from "../stream/manage.js";
 import { apiSchema } from "./schema.js";
+import { createProxyTunnels, createStream } from "../stream/manage.js";
 
 export function createResponse(responseType, responseData) {
     const internalError = (code) => {
@@ -49,6 +50,41 @@ export function createResponse(responseType, responseData) {
                 }
                 break;
 
+            case "local-processing":
+                response = {
+                    type: responseData?.type,
+                    service: responseData?.service,
+                    tunnel: createProxyTunnels(responseData),
+
+                    output: {
+                        type: mime.getType(responseData?.filename) || undefined,
+                        filename: responseData?.filename,
+                        metadata: responseData?.fileMetadata || undefined,
+                    },
+
+                    audio: {
+                        copy: responseData?.audioCopy,
+                        format: responseData?.audioFormat,
+                        bitrate: responseData?.audioBitrate,
+                    },
+
+                    isHLS: responseData?.isHLS,
+                }
+
+                if (!response.audio.format) {
+                    if (response.type === "audio") {
+                        // audio response without a format is invalid
+                        return internalError();
+                    }
+                    delete response.audio;
+                }
+
+                if (!response.output.type || !response.output.filename) {
+                    // response without a type or filename is invalid
+                    return internalError();
+                }
+                break;
+
             case "picker":
                 response = {
                     picker: responseData?.picker,
@@ -82,14 +118,13 @@ export function normalizeRequest(request) {
     ));
 }
 
-export function getIP(req) {
+export function getIP(req, prefix = 56) {
     const strippedIP = req.ip.replace(/^::ffff:/, '');
     const ip = ipaddr.parse(strippedIP);
     if (ip.kind() === 'ipv4') {
         return strippedIP;
     }
 
-    const prefix = 56;
     const v6Bytes = ip.toByteArray();
           v6Bytes.fill(0, prefix / 8);
 

api/src/processing/schema.js

@@ -20,7 +20,7 @@ export const apiSchema = z.object({
 
     filenameStyle: z.enum(
         ["classic", "pretty", "basic", "nerdy"]
-    ).default("classic"),
+    ).default("basic"),
 
     youtubeVideoCodec: z.enum(
         ["h264", "av1", "vp9"]
@@ -36,16 +36,20 @@ export const apiSchema = z.object({
                      .regex(/^[0-9a-zA-Z\-]+$/)
                      .optional(),
 
-    // TODO: remove this variable as it's no longer used
+    disableMetadata: z.boolean().default(false),
-    // and is kept for schema compatibility reasons
+
-    youtubeDubBrowserLang: z.boolean().default(false),
+    allowH265: z.boolean().default(false),
+    convertGif: z.boolean().default(true),
+    tiktokFullAudio: z.boolean().default(false),
 
     alwaysProxy: z.boolean().default(false),
-    disableMetadata: z.boolean().default(false),
+    localProcessing: z.boolean().default(false),
-    tiktokFullAudio: z.boolean().default(false),
-    tiktokH265: z.boolean().default(false),
-    twitterGif: z.boolean().default(true),
 
     youtubeHLS: z.boolean().default(false),
+    youtubeBetterAudio: z.boolean().default(false),
+
+    // temporarily kept for backwards compatibility with cobalt 10 schema
+    twitterGif: z.boolean().default(false),
+    tiktokH265: z.boolean().default(false),
 })
 .strict();

api/src/processing/service-config.js

@@ -35,13 +35,25 @@ export const services = {
     },
     instagram: {
         patterns: [
-            "reels/:postId",
-            ":username/reel/:postId",
-            "reel/:postId",
             "p/:postId",
-            ":username/p/:postId",
             "tv/:postId",
-            "stories/:username/:storyId"
+            "reel/:postId",
+            "reels/:postId",
+            "stories/:username/:storyId",
+
+            /*
+                share & username links use the same url pattern,
+                so we test the share pattern first, cuz id type is different.
+                however, if someone has the "share" username and the user
+                somehow gets a link of this ancient style, it's joever.
+            */
+
+            "share/:shareId",
+            "share/p/:shareId",
+            "share/reel/:shareId",
+
+            ":username/p/:postId",
+            ":username/reel/:postId",
         ],
         altDomains: ["ddinstagram.com"],
     },
@@ -64,8 +76,23 @@ export const services = {
     },
     reddit: {
         patterns: [
+            "comments/:id",
+
+            "r/:sub/comments/:id",
             "r/:sub/comments/:id/:title",
-            "user/:user/comments/:id/:title"
+            "r/:sub/comments/:id/comment/:commentId",
+
+            "user/:user/comments/:id",
+            "user/:user/comments/:id/:title",
+            "user/:user/comments/:id/comment/:commentId",
+
+            "r/u_:user/comments/:id",
+            "r/u_:user/comments/:id/:title",
+            "r/u_:user/comments/:id/comment/:commentId",
+
+            "r/:sub/s/:shareId",
+
+            "video/:shortId",
         ],
         subdomains: "*",
     },
@@ -89,6 +116,7 @@ export const services = {
             "add/:username",
             "u/:username",
             "t/:shortLink",
+            "o/:spotlightId",
         ],
         subdomains: ["t", "story"],
     },
@@ -111,12 +139,13 @@ export const services = {
     tiktok: {
         patterns: [
             ":user/video/:postId",
+            "i18n/share/video/:postId",
             ":shortLink",
             "t/:shortLink",
             ":user/photo/:postId",
             "v/:postId.html"
         ],
-        subdomains: ["vt", "vm", "m"],
+        subdomains: ["vt", "vm", "m", "t"],
     },
     tumblr: {
         patterns: [
@@ -166,6 +195,14 @@ export const services = {
         subdomains: ["m"],
         altDomains: ["vkvideo.ru", "vk.ru"],
     },
+    xiaohongshu: {
+        patterns: [
+            "explore/:id?xsec_token=:token",
+            "discovery/item/:id?xsec_token=:token",
+            "a/:shareId"
+        ],
+        altDomains: ["xhslink.com"],
+    },
     youtube: {
         patterns: [
             "watch?v=:id",

api/src/processing/service-patterns.js

@@ -6,7 +6,8 @@ export const testers = {
     "dailymotion": pattern => pattern.id?.length <= 32,
 
     "instagram": pattern =>
-        pattern.postId?.length <= 12
+        pattern.postId?.length <= 48
+        || pattern.shareId?.length <= 16
         || (pattern.username?.length <= 30 && pattern.storyId?.length <= 24),
 
     "loom": pattern =>
@@ -19,8 +20,11 @@ export const testers = {
         pattern.id?.length <= 128 || pattern.shortLink?.length <= 32,
 
     "reddit": pattern =>
-        (pattern.sub?.length <= 22 && pattern.id?.length <= 10)
+        pattern.id?.length <= 16 && !pattern.sub && !pattern.user
-        || (pattern.user?.length <= 22 && pattern.id?.length <= 10),
+        || (pattern.sub?.length <= 22 && pattern.id?.length <= 16)
+        || (pattern.user?.length <= 22 && pattern.id?.length <= 16)
+        || (pattern.sub?.length <= 22 && pattern.shareId?.length <= 16)
+        || (pattern.shortId?.length <= 16),
 
     "rutube": pattern =>
         (pattern.id?.length === 32 && pattern.key?.length <= 32) ||
@@ -71,4 +75,8 @@ export const testers = {
 
     "bsky": pattern =>
         pattern.user?.length <= 128 && pattern.post?.length <= 128,
+
+    "xiaohongshu": pattern =>
+        pattern.id?.length <= 24 && pattern.token?.length <= 64
+        || pattern.shareId?.length <= 24,
 }

api/src/processing/services/bilibili.js

@@ -1,19 +1,8 @@
 import { genericUserAgent, env } from "../../config.js";
+import { resolveRedirectingURL } from "../url.js";
 
 // TO-DO: higher quality downloads (currently requires an account)
 
-function com_resolveShortlink(shortId) {
-    return fetch(`https://b23.tv/${shortId}`, { redirect: 'manual' })
-            .then(r => r.status > 300 && r.status < 400 && r.headers.get('location'))
-            .then(url => {
-                if (!url) return;
-                const path = new URL(url).pathname;
-                if (path.startsWith('/video/'))
-                    return path.split('/')[2];
-            })
-            .catch(() => {})
-}
-
 function getBest(content) {
     return content?.filter(v => v.baseUrl || v.url)
                 .map(v => (v.baseUrl = v.baseUrl || v.url, v))
@@ -53,9 +42,7 @@ async function com_download(id) {
     const [ video, audio ] = extractBestQuality(streamData.data.dash);
     if (!video || !audio) {
         return { error: "fetch.empty" };
-    }
+    }    return {
-
-    return {
         urls: [video.baseUrl, audio.baseUrl],
         audioFilename: `bilibili_${id}_audio`,
         filename: `bilibili_${id}_${video.width}x${video.height}.mp4`
@@ -99,7 +86,8 @@ async function tv_download(id) {
 
 export default async function({ comId, tvId, comShortLink }) {
     if (comShortLink) {
-        comId = await com_resolveShortlink(comShortLink);
+        const patternMatch = await resolveRedirectingURL(`https://b23.tv/${comShortLink}`);
+        comId = patternMatch?.comId;
     }
 
     if (comId) {

api/src/processing/services/bluesky.js

@@ -71,6 +71,24 @@ const extractImages = ({ getPost, filename, alwaysProxy }) => {
     return { picker };
 }
 
+const extractGif = ({ url, filename }) => {
+    const gifUrl = new URL(url);
+
+    if (!gifUrl || gifUrl.hostname !== "media.tenor.com") {
+        return { error: "fetch.empty" };
+    }
+
+    // remove downscaling params from gif url
+    // such as "?hh=498&ww=498"
+    gifUrl.search = "";
+
+    return {
+        urls: gifUrl,
+        isPhoto: true,
+        filename: `${filename}.gif`,
+    }
+}
+
 export default async function ({ user, post, alwaysProxy, dispatcher }) {
     const apiEndpoint = new URL("https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread?depth=0&parentHeight=0");
     apiEndpoint.searchParams.set(
@@ -102,22 +120,37 @@ export default async function ({ user, post, alwaysProxy, dispatcher }) {
     const embedType = getPost?.thread?.post?.embed?.$type;
     const filename = `bluesky_${user}_${post}`;
 
-    if (embedType === "app.bsky.embed.video#view") {
+    switch (embedType) {
+        case "app.bsky.embed.video#view":
             return extractVideo({
                 media: getPost.thread?.post?.embed,
                 filename,
-        })
+            });
-    }
 
-    if (embedType === "app.bsky.embed.recordWithMedia#view") {
+        case "app.bsky.embed.images#view":
+            return extractImages({
+                getPost,
+                filename,
+                alwaysProxy
+            });
+
+        case "app.bsky.embed.external#view":
+            return extractGif({
+                url: getPost?.thread?.post?.embed?.external?.uri,
+                filename,
+            });
+
+        case "app.bsky.embed.recordWithMedia#view":
+            if (getPost?.thread?.post?.embed?.media?.$type === "app.bsky.embed.external#view") {
+                return extractGif({
+                    url: getPost?.thread?.post?.embed?.media?.external?.uri,
+                    filename,
+                });
+            }
             return extractVideo({
                 media: getPost.thread?.post?.embed?.media,
                 filename,
-        })
+            });
-    }
-
-    if (embedType === "app.bsky.embed.images#view") {
-        return extractImages({ getPost, filename, alwaysProxy });
     }
 
     return { error: "fetch.empty" };

api/src/processing/services/facebook.js

@@ -8,8 +8,8 @@ const headers = {
     'Sec-Fetch-Site': 'none',
 }
 
-const resolveUrl = (url) => {
+const resolveUrl = (url, dispatcher) => {
-    return fetch(url, { headers })
+    return fetch(url, { headers, dispatcher })
         .then(r => {
             if (r.headers.get('location')) {
                 return decodeURIComponent(r.headers.get('location'));
@@ -23,13 +23,13 @@ const resolveUrl = (url) => {
         .catch(() => false);
 }
 
-export default async function({ id, shareType, shortLink }) {
+export default async function({ id, shareType, shortLink, dispatcher }) {
     let url = `https://web.facebook.com/i/videos/${id}`;
 
     if (shareType) url = `https://web.facebook.com/share/${shareType}/${id}`;
-    if (shortLink) url = await resolveUrl(`https://fb.watch/${shortLink}`);
+    if (shortLink) url = await resolveUrl(`https://fb.watch/${shortLink}`, dispatcher);
 
-    const html = await fetch(url, { headers })
+    const html = await fetch(url, { headers, dispatcher })
         .then(r => r.text())
         .catch(() => false);
 

api/src/processing/services/instagram.js

@@ -1,3 +1,5 @@
+import { randomBytes } from "node:crypto";
+import { resolveRedirectingURL } from "../url.js";
 import { genericUserAgent } from "../../config.js";
 import { createStream } from "../../stream/manage.js";
 import { getCookie, updateCookie } from "../cookie/manager.js";
@@ -8,6 +10,7 @@ const commonHeaders = {
     "sec-fetch-site": "same-origin",
     "x-ig-app-id": "936619743392459"
 }
+
 const mobileHeaders = {
     "x-ig-app-locale": "en_US",
     "x-ig-device-locale": "en_US",
@@ -19,6 +22,7 @@ const mobileHeaders = {
     "x-fb-server-cluster": "True",
     "content-length": "0",
 }
+
 const embedHeaders = {
     "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
     "Accept-Language": "en-GB,en;q=0.9",
@@ -33,7 +37,7 @@ const embedHeaders = {
     "Sec-Fetch-Site": "none",
     "Sec-Fetch-User": "?1",
     "Upgrade-Insecure-Requests": "1",
-    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
+    "User-Agent": genericUserAgent,
 }
 
 const cachedDtsg = {
@@ -41,7 +45,17 @@ const cachedDtsg = {
     expiry: 0
 }
 
-export default function(obj) {
+const getNumberFromQuery = (name, data) => {
+    const s = data?.match(new RegExp(name + '=(\\d+)'))?.[1];
+    if (+s) return +s;
+}
+
+const getObjectFromEntries = (name, data) => {
+    const obj = data?.match(new RegExp('\\["' + name + '",.*?,({.*?}),\\d+\\]'))?.[1];
+    return obj && JSON.parse(obj);
+}
+
+export default function instagram(obj) {
     const dispatcher = obj.dispatcher;
 
     async function findDtsgId(cookie) {
@@ -91,6 +105,7 @@ export default function(obj) {
         updateCookie(cookie, data.headers);
         return data.json();
     }
+
     async function getMediaId(id, { cookie, token } = {}) {
         const oembedURL = new URL('https://i.instagram.com/api/v1/oembed/');
         oembedURL.searchParams.set('url', `https://www.instagram.com/p/${id}/`);
@@ -119,6 +134,7 @@ export default function(obj) {
 
         return mediaInfo?.items?.[0];
     }
+
     async function requestHTML(id, cookie) {
         const data = await fetch(`https://www.instagram.com/p/${id}/embed/captioned/`, {
             headers: {
@@ -136,40 +152,167 @@ export default function(obj) {
 
         return embedData;
     }
-    async function requestGQL(id, cookie) {
-        let dtsgId;
 
-        if (cookie) {
+    async function getGQLParams(id, cookie) {
-            dtsgId = await findDtsgId(cookie);
+        const req = await fetch(`https://www.instagram.com/p/${id}/`, {
+            headers: {
+                ...embedHeaders,
+                cookie
+            },
+            dispatcher
+        });
+
+        const html = await req.text();
+        const siteData = getObjectFromEntries('SiteData', html);
+        const polarisSiteData = getObjectFromEntries('PolarisSiteData', html);
+        const webConfig = getObjectFromEntries('DGWWebConfig', html);
+        const pushInfo = getObjectFromEntries('InstagramWebPushInfo', html);
+        const lsd = getObjectFromEntries('LSD', html)?.token || randomBytes(8).toString('base64url');
+        const csrf = getObjectFromEntries('InstagramSecurityConfig', html)?.csrf_token;
+
+        const anon_cookie = [
+            csrf && "csrftoken=" + csrf,
+            polarisSiteData?.device_id && "ig_did=" + polarisSiteData?.device_id,
+            "wd=1280x720",
+            "dpr=2",
+            polarisSiteData?.machine_id && "mid=" + polarisSiteData.machine_id,
+            "ig_nrcb=1"
+        ].filter(a => a).join('; ');
+
+        return {
+            headers: {
+                'x-ig-app-id': webConfig?.appId || '936619743392459',
+                'X-FB-LSD': lsd,
+                'X-CSRFToken': csrf,
+                'X-Bloks-Version-Id': getObjectFromEntries('WebBloksVersioningID', html)?.versioningID,
+                'x-asbd-id': 129477,
+                cookie: anon_cookie
+            },
+            body: {
+                __d: 'www',
+                __a: '1',
+                __s: '::' + Math.random().toString(36).substring(2).replace(/\d/g, '').slice(0, 6),
+                __hs: siteData?.haste_session || '20126.HYP:instagram_web_pkg.2.1...0',
+                __req: 'b',
+                __ccg: 'EXCELLENT',
+                __rev: pushInfo?.rollout_hash || '1019933358',
+                __hsi: siteData?.hsi || '7436540909012459023',
+                __dyn: randomBytes(154).toString('base64url'),
+                __csr: randomBytes(154).toString('base64url'),
+                __user: '0',
+                __comet_req: getNumberFromQuery('__comet_req', html) || '7',
+                av: '0',
+                dpr: '2',
+                lsd,
+                jazoest: getNumberFromQuery('jazoest', html) || Math.floor(Math.random() * 10000),
+                __spin_r: siteData?.__spin_r || '1019933358',
+                __spin_b: siteData?.__spin_b || 'trunk',
+                __spin_t: siteData?.__spin_t || Math.floor(new Date().getTime() / 1000),
+            }
+        };
     }
-        const url = new URL('https://www.instagram.com/api/graphql/');
 
-        const requestData = {
+    async function requestGQL(id, cookie) {
-            jazoest: '26406',
+        const { headers, body } = await getGQLParams(id, cookie);
+
+        const req = await fetch('https://www.instagram.com/graphql/query', {
+            method: 'POST',
+            dispatcher,
+            headers: {
+                ...embedHeaders,
+                ...headers,
+                cookie,
+                'content-type': 'application/x-www-form-urlencoded',
+                'X-FB-Friendly-Name': 'PolarisPostActionLoadPostQueryQuery',
+            },
+            body: new URLSearchParams({
+                ...body,
+                fb_api_caller_class: 'RelayModern',
+                fb_api_req_friendly_name: 'PolarisPostActionLoadPostQueryQuery',
                 variables: JSON.stringify({
                     shortcode: id,
-            __relay_internal__pv__PolarisShareMenurelayprovider: false
+                    fetch_tagged_user_count: null,
+                    hoisted_comment_id: null,
+                    hoisted_reply_id: null
                 }),
-            doc_id: '7153618348081770'
+                server_timestamps: true,
+                doc_id: '8845758582119845'
+            }).toString()
+        });
+
+        return {
+            gql_data: await req.json()
+                        .then(r => r.data)
+                        .catch(() => null)
         };
-        if (dtsgId) {
-            requestData.fb_dtsg = dtsgId;
     }
 
-        return (await request(url, cookie, 'POST', requestData))
+    async function getErrorContext(id) {
-                .data
+        try {
-                ?.xdt_api__v1__media__shortcode__web_info
+            const { headers, body } = await getGQLParams(id);
-                ?.items
+
-                ?.[0];
+            const req = await fetch('https://www.instagram.com/ajax/bulk-route-definitions/', {
+                method: 'POST',
+                dispatcher,
+                headers: {
+                    ...embedHeaders,
+                    ...headers,
+                    'content-type': 'application/x-www-form-urlencoded',
+                    'X-Ig-D': 'www',
+                },
+                body: new URLSearchParams({
+                    'route_urls[0]': `/p/${id}/`,
+                    routing_namespace: 'igx_www',
+                    ...body
+                }).toString()
+            });
+
+            const response = await req.text();
+            if (response.includes('"tracePolicy":"polaris.privatePostPage"'))
+                return { error: 'content.post.private' };
+
+            const [, mediaId, mediaOwnerId] = response.match(
+                /"media_id":\s*?"(\d+)","media_owner_id":\s*?"(\d+)"/
+            ) || [];
+
+            if (mediaId && mediaOwnerId) {
+                const rulingURL = new URL('https://www.instagram.com/api/v1/web/get_ruling_for_media_content_logged_out');
+                rulingURL.searchParams.set('media_id', mediaId);
+                rulingURL.searchParams.set('owner_id', mediaOwnerId);
+
+                const rulingResponse = await fetch(rulingURL, {
+                    headers: {
+                        ...headers,
+                        ...commonHeaders
+                    },
+                    dispatcher,
+                }).then(a => a.json()).catch(() => ({}));
+
+                if (rulingResponse?.title?.includes('Restricted'))
+                    return { error: "content.post.age" };
+            }
+        } catch {
+            return { error: "fetch.fail" };
+        }
+
+        return { error: "fetch.empty" };
     }
 
     function extractOldPost(data, id, alwaysProxy) {
-        const sidecar = data?.gql_data?.shortcode_media?.edge_sidecar_to_children;
+        const shortcodeMedia = data?.gql_data?.shortcode_media || data?.gql_data?.xdt_shortcode_media;
+        const sidecar = shortcodeMedia?.edge_sidecar_to_children;
+
         if (sidecar) {
             const picker = sidecar.edges.filter(e => e.node?.display_url)
                 .map((e, i) => {
-                    const type = e.node?.is_video ? "video" : "photo";
+                    const type = e.node?.is_video && e.node?.video_url ? "video" : "photo";
-                    const url = type === "video" ? e.node?.video_url : e.node?.display_url;
+
+                    let url;
+                    if (type === "video") {
+                        url = e.node?.video_url;
+                    } else if (type === "photo") {
+                        url = e.node?.display_url;
+                    }
 
                     let itemExt = type === "video" ? "mp4" : "jpg";
 
@@ -196,16 +339,21 @@ export default function(obj) {
                 });
 
             if (picker.length) return { picker }
-        } else if (data?.gql_data?.shortcode_media?.video_url) {
+        }
+
+        if (shortcodeMedia?.video_url) {
             return {
-                urls: data.gql_data.shortcode_media.video_url,
+                urls: shortcodeMedia.video_url,
                 filename: `instagram_${id}.mp4`,
                 audioFilename: `instagram_${id}_audio`
             }
-        } else if (data?.gql_data?.shortcode_media?.display_url) {
+        }
+
+        if (shortcodeMedia?.display_url) {
             return {
-                urls: data.gql_data?.shortcode_media.display_url,
+                urls: shortcodeMedia.display_url,
-                isPhoto: true
+                isPhoto: true,
+                filename: `instagram_${id}.jpg`,
             }
         }
     }
@@ -266,7 +414,9 @@ export default function(obj) {
     }
 
     async function getPost(id, alwaysProxy) {
-        const hasData = (data) => data && data.gql_data !== null;
+        const hasData = (data) => data
+                                    && data.gql_data !== null
+                                    && data?.gql_data?.xdt_shortcode_media !== null;
         let data, result;
         try {
             const cookie = getCookie('instagram');
@@ -295,7 +445,9 @@ export default function(obj) {
             if (!hasData(data) && cookie) data = await requestGQL(id, cookie);
         } catch {}
 
-        if (!data) return { error: "fetch.fail" };
+        if (!hasData(data)) {
+            return getErrorContext(id);
+        }
 
         if (data?.gql_data) {
             result = extractOldPost(data, id, alwaysProxy)
@@ -358,14 +510,30 @@ export default function(obj) {
         if (item.image_versions2?.candidates) {
             return {
                 urls: item.image_versions2.candidates[0].url,
-                isPhoto: true
+                isPhoto: true,
+                filename: `instagram_${id}.jpg`,
             }
         }
 
         return { error: "link.unsupported" };
     }
 
-    const { postId, storyId, username, alwaysProxy } = obj;
+    const { postId, shareId, storyId, username, alwaysProxy } = obj;
+
+    if (shareId) {
+        return resolveRedirectingURL(
+            `https://www.instagram.com/share/${shareId}/`,
+            dispatcher,
+            // for some reason instagram decides to return HTML
+            // instead of a redirect when requesting with a normal
+            // browser user-agent
+            {'User-Agent': 'curl/7.88.1'}
+        ).then(match => instagram({
+            ...obj, ...match,
+            shareId: undefined
+        }));
+    }
+
     if (postId) return getPost(postId, alwaysProxy);
     if (username && storyId) return getStory(username, storyId);
 

api/src/processing/services/loom.js

@@ -1,18 +1,18 @@
 import { genericUserAgent } from "../../config.js";
 
-export default async function({ id }) {
+const craftHeaders = id => ({
-    const gql = await fetch(`https://www.loom.com/api/campaigns/sessions/${id}/transcoded-url`, {
-        method: "POST",
-        headers: {
     "user-agent": genericUserAgent,
+    "content-type": "application/json",
     origin: "https://www.loom.com",
     referer: `https://www.loom.com/share/${id}`,
     cookie: `loom_referral_video=${id};`,
+    "x-loom-request-source": "loom_web_be851af",
+});
 
-            "apollographql-client-name": "web",
+async function fromTranscodedURL(id) {
-            "apollographql-client-version": "14c0b42",
+    const gql = await fetch(`https://www.loom.com/api/campaigns/sessions/${id}/transcoded-url`, {
-            "x-loom-request-source": "loom_web_14c0b42",
+        method: "POST",
-        },
+        headers: craftHeaders(id),
         body: JSON.stringify({
             force_original: false,
             password: null,
@@ -20,20 +20,47 @@ export default async function({ id }) {
             deviceID: null
         })
     })
-    .then(r => r.status === 200 ? r.json() : false)
+    .then(r => r.status === 200 && r.json())
     .catch(() => {});
 
-    if (!gql) return { error: "fetch.empty" };
+    if (gql?.url?.includes('.mp4?')) {
+        return gql.url;
+    }
+}
 
-    const videoUrl = gql?.url;
+async function fromRawURL(id) {
+    const gql = await fetch(`https://www.loom.com/api/campaigns/sessions/${id}/raw-url`, {
+        method: "POST",
+        headers: craftHeaders(id),
+        body: JSON.stringify({
+            anonID: crypto.randomUUID(),
+            client_name: "web",
+            client_version: "be851af",
+            deviceID: null,
+            force_original: false,
+            password: null,
+            supported_mime_types: ["video/mp4"],
+        })
+    })
+    .then(r => r.status === 200 && r.json())
+    .catch(() => {});
+
+    if (gql?.url?.includes('.mp4?')) {
+        return gql.url;
+    }
+}
+
+export default async function({ id }) {
+    let url = await fromTranscodedURL(id);
+    url ??= await fromRawURL(id);
+
+    if (!url) {
+        return { error: "fetch.empty" }
+    }
 
-    if (videoUrl?.includes('.mp4?')) {
     return {
-            urls: videoUrl,
+        urls: url,
         filename: `loom_${id}.mp4`,
         audioFilename: `loom_${id}_audio`
     }
-    }
-
-    return { error: "fetch.empty" }
 }

api/src/processing/services/ok.js

@@ -44,7 +44,7 @@ export default async function(o) {
 
     let fileMetadata = {
         title: videoData.movie.title.trim(),
-        author: (videoData.author?.name || videoData.compilationTitle).trim(),
+        author: (videoData.author?.name || videoData.compilationTitle)?.trim(),
     }
 
     if (bestVideo) return {

api/src/processing/services/pinterest.js

@@ -1,4 +1,5 @@
 import { genericUserAgent } from "../../config.js";
+import { resolveRedirectingURL } from "../url.js";
 
 const videoRegex = /"url":"(https:\/\/v1\.pinimg\.com\/videos\/.*?)"/g;
 const imageRegex = /src="(https:\/\/i\.pinimg\.com\/.*\.(jpg|gif))"/g;
@@ -7,10 +8,10 @@ export default async function(o) {
     let id = o.id;
 
     if (!o.id && o.shortLink) {
-        id = await fetch(`https://api.pinterest.com/url_shortener/${o.shortLink}/redirect/`, { redirect: "manual" })
+        const patternMatch = await resolveRedirectingURL(`https://api.pinterest.com/url_shortener/${o.shortLink}/redirect/`);
-                   .then(r => r.headers.get("location").split('pin/')[1].split('/')[0])
+        id = patternMatch?.id;
-                   .catch(() => {});
     }
+
     if (id.includes("--")) id = id.split("--")[1];
     if (!id) return { error: "fetch.fail" };
 
@@ -22,12 +23,12 @@ export default async function(o) {
 
     const videoLink = [...html.matchAll(videoRegex)]
                     .map(([, link]) => link)
-                    .find(a => a.endsWith('.mp4') && a.includes('720p'));
+                    .find(a => a.endsWith('.mp4'));
 
     if (videoLink) return {
         urls: videoLink,
-        filename: `pinterest_${o.id}.mp4`,
+        filename: `pinterest_${id}.mp4`,
-        audioFilename: `pinterest_${o.id}_audio`
+        audioFilename: `pinterest_${id}_audio`
     }
 
     const imageLink = [...html.matchAll(imageRegex)]
@@ -39,7 +40,7 @@ export default async function(o) {
     if (imageLink) return {
         urls: imageLink,
         isPhoto: true,
-        filename: `pinterest_${o.id}.${imageType}`
+        filename: `pinterest_${id}.${imageType}`
     }
 
     return { error: "fetch.empty" };

api/src/processing/services/reddit.js

@@ -1,3 +1,4 @@
+import { resolveRedirectingURL } from "../url.js";
 import { genericUserAgent, env } from "../../config.js";
 import { getCookie, updateCookieValues } from "../cookie/manager.js";
 
@@ -48,23 +49,36 @@ async function getAccessToken() {
 }
 
 export default async function(obj) {
-    let url = new URL(`https://www.reddit.com/r/${obj.sub}/comments/${obj.id}.json`);
+    let params = obj;
+    const accessToken = await getAccessToken();
+    const headers = {
+        'user-agent': genericUserAgent,
+        authorization: accessToken && `Bearer ${accessToken}`,
+        accept: 'application/json'
+    };
 
-    if (obj.user) {
+    if (params.shortId) {
-        url.pathname = `/user/${obj.user}/comments/${obj.id}.json`;
+        params = await resolveRedirectingURL(
+            `https://www.reddit.com/video/${params.shortId}`,
+            obj.dispatcher, headers
+        );
     }
 
-    const accessToken = await getAccessToken();
+    if (!params.id && params.shareId) {
+        params = await resolveRedirectingURL(
+            `https://www.reddit.com/r/${params.sub}/s/${params.shareId}`,
+            obj.dispatcher, headers
+        );
+    }
+
+    if (!params?.id) return { error: "fetch.short_link" };
+
+    const url = new URL(`https://www.reddit.com/comments/${params.id}.json`);
+
     if (accessToken) url.hostname = 'oauth.reddit.com';
 
     let data = await fetch(
-        url, {
+        url, { headers }
-            headers: {
-                'User-Agent': genericUserAgent,
-                accept: 'application/json',
-                authorization: accessToken && `Bearer ${accessToken}`
-            }
-        }
     ).then(r => r.json()).catch(() => {});
 
     if (!data || !Array.isArray(data)) {
@@ -73,12 +87,17 @@ export default async function(obj) {
 
     data = data[0]?.data?.children[0]?.data;
 
-    const id = `${String(obj.sub).toLowerCase()}_${obj.id}`;
+    let sourceId;
+    if (params.sub || params.user) {
+        sourceId = `${String(params.sub || params.user).toLowerCase()}_${params.id}`;
+    } else {
+        sourceId = params.id;
+    }
 
     if (data?.url?.endsWith('.gif')) return {
         typeId: "redirect",
         urls: data.url,
-        filename: `reddit_${id}.gif`,
+        filename: `reddit_${sourceId}.gif`,
     }
 
     if (!data.secure_media?.reddit_video)
@@ -87,8 +106,9 @@ export default async function(obj) {
     if (data.secure_media?.reddit_video?.duration > env.durationLimit)
         return { error: "content.too_long" };
 
+    const video = data.secure_media?.reddit_video?.fallback_url?.split('?')[0];
+
     let audio = false,
-        video = data.secure_media?.reddit_video?.fallback_url?.split('?')[0],
         audioFileLink = `${data.secure_media?.reddit_video?.fallback_url?.split('DASH')[0]}audio`;
 
     if (video.match('.mp4')) {
@@ -121,7 +141,7 @@ export default async function(obj) {
         typeId: "tunnel",
         type: "merge",
         urls: [video, audioFileLink],
-        audioFilename: `reddit_${id}_audio`,
+        audioFilename: `reddit_${sourceId}_audio`,
-        filename: `reddit_${id}.mp4`
+        filename: `reddit_${sourceId}.mp4`
     }
 }

api/src/processing/services/snapchat.js

@@ -1,7 +1,6 @@
-import { extract, normalizeURL } from "../url.js";
+import { resolveRedirectingURL } from "../url.js";
 import { genericUserAgent } from "../../config.js";
 import { createStream } from "../../stream/manage.js";
-import { getRedirectingURL } from "../../misc/utils.js";
 
 const SPOTLIGHT_VIDEO_REGEX = /<link data-react-helmet="true" rel="preload" href="([^"]+)" as="video"\/>/;
 const NEXT_DATA_REGEX = /<script id="__NEXT_DATA__" type="application\/json">({.+})<\/script><\/body><\/html>$/;
@@ -41,9 +40,9 @@ async function getStory(username, storyId, alwaysProxy) {
     const nextDataString = html.match(NEXT_DATA_REGEX)?.[1];
     if (nextDataString) {
         const data = JSON.parse(nextDataString);
-        const storyIdParam = data.query.profileParams[1];
+        const storyIdParam = data?.query?.profileParams?.[1];
 
-        if (storyIdParam && data.props.pageProps.story) {
+        if (storyIdParam && data?.props?.pageProps?.story) {
             const story = data.props.pageProps.story.snapList.find((snap) => snap.snapId.value === storyIdParam);
             if (story) {
                 if (story.snapMediaType === 0) {
@@ -62,7 +61,7 @@ async function getStory(username, storyId, alwaysProxy) {
             }
         }
 
-        const defaultStory = data.props.pageProps.curatedHighlights[0];
+        const defaultStory = data?.props?.pageProps?.curatedHighlights?.[0];
         if (defaultStory) {
             return {
                 picker: defaultStory.snapList.map(snap => {
@@ -100,24 +99,13 @@ async function getStory(username, storyId, alwaysProxy) {
 export default async function (obj) {
     let params = obj;
     if (obj.shortLink) {
-        const link = await getRedirectingURL(`https://t.snapchat.com/${obj.shortLink}`);
+        params = await resolveRedirectingURL(`https://t.snapchat.com/${obj.shortLink}`);
-
-        if (!link?.startsWith('https://www.snapchat.com/')) {
-            return { error: "fetch.short_link" };
     }
 
-        const extractResult = extract(normalizeURL(link));
+    if (params?.spotlightId) {
-        if (extractResult?.host !== 'snapchat') {
-            return { error: "fetch.short_link" };
-        }
-
-        params = extractResult.patternMatch;
-    }
-
-    if (params.spotlightId) {
         const result = await getSpotlight(params.spotlightId);
         if (result) return result;
-    } else if (params.username) {
+    } else if (params?.username) {
         const result = await getStory(params.username, params.storyId, obj.alwaysProxy);
         if (result) return result;
     }

api/src/processing/services/soundcloud.js

@@ -1,4 +1,5 @@
 import { env } from "../../config.js";
+import { resolveRedirectingURL } from "../url.js";
 
 const cachedID = {
     version: '',
@@ -7,22 +8,25 @@ const cachedID = {
 
 async function findClientID() {
     try {
-        let sc = await fetch('https://soundcloud.com/').then(r => r.text()).catch(() => {});
+        const sc = await fetch('https://soundcloud.com/').then(r => r.text()).catch(() => {});
-        let scVersion = String(sc.match(/<script>window\.__sc_version="[0-9]{10}"<\/script>/)[0].match(/[0-9]{10}/));
+        const scVersion = String(sc.match(/<script>window\.__sc_version="[0-9]{10}"<\/script>/)[0].match(/[0-9]{10}/));
 
-        if (cachedID.version === scVersion) return cachedID.id;
+        if (cachedID.version === scVersion) {
+            return cachedID.id;
+        }
+
+        const scripts = sc.matchAll(/<script.+src="(.+)">/g);
 
-        let scripts = sc.matchAll(/<script.+src="(.+)">/g);
         let clientid;
         for (let script of scripts) {
-            let url = script[1];
+            const url = script[1];
 
             if (!url?.startsWith('https://a-v2.sndcdn.com/')) {
                 return;
             }
 
-            let scrf = await fetch(url).then(r => r.text()).catch(() => {});
+            const scrf = await fetch(url).then(r => r.text()).catch(() => {});
-            let id = scrf.match(/\("client_id=[A-Za-z0-9]{32}"\)/);
+            const id = scrf.match(/\("client_id=[A-Za-z0-9]{32}"\)/);
 
             if (id && typeof id[0] === 'string') {
                 clientid = id[0].match(/[A-Za-z0-9]{32}/)[0];
@@ -37,46 +41,57 @@ async function findClientID() {
 }
 
 export default async function(obj) {
-    let clientId = await findClientID();
+    const clientId = await findClientID();
     if (!clientId) return { error: "fetch.fail" };
 
     let link;
-    if (obj.url.hostname === 'on.soundcloud.com' && obj.shortLink) {
+
-        link = await fetch(`https://on.soundcloud.com/${obj.shortLink}/`, { redirect: "manual" }).then(r => {
+    if (obj.shortLink) {
-            if (r.status === 302 && r.headers.get("location").startsWith("https://soundcloud.com/")) {
+        obj = {
-                return r.headers.get("location").split('?', 1)[0]
+            ...obj,
+            ...await resolveRedirectingURL(
+                `https://on.soundcloud.com/${obj.shortLink}`
+            )
         }
-        }).catch(() => {});
     }
 
-    if (!link && obj.author && obj.song) {
+    if (obj.author && obj.song) {
-        link = `https://soundcloud.com/${obj.author}/${obj.song}${obj.accessKey ? `/s-${obj.accessKey}` : ''}`
+        link = `https://soundcloud.com/${obj.author}/${obj.song}`;
+        if (obj.accessKey) {
+            link += `/s-${obj.accessKey}`;
+        }
     }
 
     if (!link && obj.shortLink) return { error: "fetch.short_link" };
     if (!link) return { error: "link.unsupported" };
 
-    let json = await fetch(`https://api-v2.soundcloud.com/resolve?url=${link}&client_id=${clientId}`)
+    const resolveURL = new URL("https://api-v2.soundcloud.com/resolve");
-                     .then(r => r.status === 200 ? r.json() : false)
+    resolveURL.searchParams.set("url", link);
-                     .catch(() => {});
+    resolveURL.searchParams.set("client_id", clientId);
 
+    const json = await fetch(resolveURL).then(r => r.json()).catch(() => {});
     if (!json) return { error: "fetch.fail" };
 
-    if (json?.policy === "BLOCK") {
+    if (json.duration > env.durationLimit * 1000) {
+        return { error: "content.too_long" };
+    }
+
+    if (json.policy === "BLOCK") {
         return { error: "content.region" };
     }
 
-    if (json?.policy === "SNIP") {
+    if (json.policy === "SNIP") {
         return { error: "content.paid" };
     }
 
-    if (!json?.media?.transcodings || !json?.media?.transcodings.length === 0) {
+    if (!json.media?.transcodings || !json.media?.transcodings.length === 0) {
         return { error: "fetch.empty" };
     }
 
     let bestAudio = "opus",
-        selectedStream = json.media.transcodings.find(v => v.preset === "opus_0_0"),
+        selectedStream = json.media.transcodings.find(v => v.preset === "opus_0_0");
-        mp3Media = json.media.transcodings.find(v => v.preset === "mp3_0_0");
+
+    const mp3Media = json.media.transcodings.find(v => v.preset === "mp3_0_0");
 
     // use mp3 if present if user prefers it or if opus isn't available
     if (mp3Media && (obj.format === "mp3" || !selectedStream)) {
@@ -88,35 +103,30 @@ export default async function(obj) {
         return { error: "fetch.empty" };
     }
 
-    let fileUrlBase = selectedStream.url;
+    const fileUrl = new URL(selectedStream.url);
-    let fileUrl = `${fileUrlBase}${fileUrlBase.includes("?") ? "&" : "?"}client_id=${clientId}&track_authorization=${json.track_authorization}`;
+    fileUrl.searchParams.set("client_id", clientId);
+    fileUrl.searchParams.set("track_authorization", json.track_authorization);
 
-    if (!fileUrl.startsWith("https://api-v2.soundcloud.com/media/soundcloud:tracks:"))
+    const file = await fetch(fileUrl)
-        return { error: "fetch.empty" };
+                     .then(async r => new URL((await r.json()).url))
-
-    if (json.duration > env.durationLimit * 1000) {
-        return { error: "content.too_long" };
-    }
-
-    let file = await fetch(fileUrl)
-                     .then(async r => (await r.json()).url)
                      .catch(() => {});
+
     if (!file) return { error: "fetch.empty" };
 
-    let fileMetadata = {
+    const fileMetadata = {
         title: json.title.trim(),
         artist: json.user.username.trim(),
     }
 
     return {
-        urls: file,
+        urls: file.toString(),
         filenameAttributes: {
             service: "soundcloud",
             id: json.id,
-            title: fileMetadata.title,
+            ...fileMetadata
-            author: fileMetadata.artist
         },
         bestAudio,
-        fileMetadata
+        fileMetadata,
+        isHLS: file.pathname.endsWith('.m3u8'),
     }
 }

api/src/processing/services/tiktok.js

@@ -1,6 +1,6 @@
 import Cookie from "../cookie/cookie.js";
 
-import { extract } from "../url.js";
+import { extract, normalizeURL } from "../url.js";
 import { genericUserAgent } from "../../config.js";
 import { updateCookie } from "../cookie/manager.js";
 import { createStream } from "../../stream/manage.js";
@@ -23,14 +23,14 @@ export default async function(obj) {
 
         if (html.startsWith('<a href="https://')) {
             const extractedURL = html.split('<a href="')[1].split('?')[0];
-            const { patternMatch } = extract(extractedURL);
+            const { patternMatch } = extract(normalizeURL(extractedURL));
-            postId = patternMatch.postId;
+            postId = patternMatch?.postId;
         }
     }
     if (!postId) return { error: "fetch.short_link" };
 
     // should always be /video/, even for photos
-    const res = await fetch(`https://tiktok.com/@i/video/${postId}`, {
+    const res = await fetch(`https://www.tiktok.com/@i/video/${postId}`, {
         headers: {
             "user-agent": genericUserAgent,
             cookie,

api/src/processing/services/twitter.js

@@ -24,6 +24,11 @@ const badContainerEnd = new Date(1702605600000);
 
 function needsFixing(media) {
     const representativeId = media.source_status_id_str ?? media.id_str;
+
+    // syndication api doesn't have media ids in its response,
+    // so we just assume it's all good
+    if (!representativeId) return false;
+
     const mediaTimestamp = new Date(
         Number((BigInt(representativeId) >> 22n) + TWITTER_EPOCH)
     );
@@ -53,6 +58,25 @@ const getGuestToken = async (dispatcher, forceReload = false) => {
     }
 }
 
+const requestSyndication = async(dispatcher, tweetId) => {
+    // thank you
+    // https://github.com/yt-dlp/yt-dlp/blob/05c8023a27dd37c49163c0498bf98e3e3c1cb4b9/yt_dlp/extractor/twitter.py#L1334
+    const token = (id) => ((Number(id) / 1e15) * Math.PI).toString(36).replace(/(0+|\.)/g, '');
+    const syndicationUrl = new URL("https://cdn.syndication.twimg.com/tweet-result");
+
+    syndicationUrl.searchParams.set("id", tweetId);
+    syndicationUrl.searchParams.set("token", token(tweetId));
+
+    const result = await fetch(syndicationUrl, {
+        headers: {
+            "user-agent": genericUserAgent
+        },
+        dispatcher
+    });
+
+    return result;
+}
+
 const requestTweet = async(dispatcher, tweetId, token, cookie) => {
     const graphqlTweetURL = new URL(graphqlURL);
 
@@ -87,36 +111,24 @@ const requestTweet = async(dispatcher, tweetId, token, cookie) => {
     let result = await fetch(graphqlTweetURL, { headers, dispatcher });
     updateCookie(cookie, result.headers);
 
-    // we might have been missing the `ct0` cookie, retry
+    // we might have been missing the ct0 cookie, retry
     if (result.status === 403 && result.headers.get('set-cookie')) {
+        const cookieValues = cookie?.values();
+        if (cookieValues?.ct0) {
             result = await fetch(graphqlTweetURL, {
                 headers: {
                     ...headers,
-                'x-csrf-token': cookie.values().ct0
+                    'x-csrf-token': cookieValues.ct0
                 },
                 dispatcher
             });
         }
+    }
 
     return result
 }
 
-export default async function({ id, index, toGif, dispatcher, alwaysProxy }) {
+const extractGraphqlMedia = async (tweet, dispatcher, id, guestToken, cookie) => {
-    const cookie = await getCookie('twitter');
-
-    let guestToken = await getGuestToken(dispatcher);
-    if (!guestToken) return { error: "fetch.fail" };
-
-    let tweet = await requestTweet(dispatcher, id, guestToken);
-
-    // get new token & retry if old one expired
-    if ([403, 429].includes(tweet.status)) {
-        guestToken = await getGuestToken(dispatcher, true);
-        tweet = await requestTweet(dispatcher, id, guestToken)
-    }
-
-    tweet = await tweet.json();
-
     let tweetTypename = tweet?.data?.tweetResult?.result?.__typename;
 
     if (!tweetTypename) {
@@ -127,13 +139,13 @@ export default async function({ id, index, toGif, dispatcher, alwaysProxy }) {
         const reason = tweet?.data?.tweetResult?.result?.reason;
         switch(reason) {
             case "Protected":
-                return { error: "content.post.private" }
+                return { error: "content.post.private" };
             case "NsfwLoggedOut":
                 if (cookie) {
                     tweet = await requestTweet(dispatcher, id, guestToken, cookie);
                     tweet = await tweet.json();
                     tweetTypename = tweet?.data?.tweetResult?.result?.__typename;
-                } else return { error: "content.post.age" }
+                } else return { error: "content.post.age" };
         }
     }
 
@@ -150,7 +162,82 @@ export default async function({ id, index, toGif, dispatcher, alwaysProxy }) {
         repostedTweet = baseTweet?.retweeted_status_result?.result.tweet.legacy.extended_entities;
     }
 
-    let media = (repostedTweet?.media || baseTweet?.extended_entities?.media);
+    if (tweetResult.card?.legacy?.binding_values?.length) {
+        const card = JSON.parse(tweetResult.card.legacy.binding_values[0].value?.string_value);
+
+        if (!["video_website", "image_website"].includes(card?.type) ||
+            !card?.media_entities ||
+            card?.component_objects?.media_1?.type !== "media") {
+            return;
+        }
+
+        const mediaId = card.component_objects?.media_1?.data?.id;
+        return [card.media_entities[mediaId]];
+    }
+
+    return (repostedTweet?.media || baseTweet?.extended_entities?.media);
+}
+
+const testResponse = (result) => {
+    const contentLength = result.headers.get("content-length");
+
+    if (!contentLength || contentLength === '0') {
+        return false;
+    }
+
+    if (!result.headers.get("content-type").startsWith("application/json")) {
+        return false;
+    }
+
+    return true;
+}
+
+export default async function({ id, index, toGif, dispatcher, alwaysProxy }) {
+    const cookie = await getCookie('twitter');
+
+    let syndication = false;
+
+    let guestToken = await getGuestToken(dispatcher);
+    if (!guestToken) return { error: "fetch.fail" };
+
+    // for now we assume that graphql api will come back after some time,
+    // so we try it first
+
+    let tweet = await requestTweet(dispatcher, id, guestToken);
+
+    // get new token & retry if old one expired
+    if ([403, 429].includes(tweet.status)) {
+        guestToken = await getGuestToken(dispatcher, true);
+        if (cookie) {
+            tweet = await requestTweet(dispatcher, id, guestToken, cookie);
+        } else {
+            tweet = await requestTweet(dispatcher, id, guestToken);
+        }
+    }
+
+    const testGraphql = testResponse(tweet);
+
+    // if graphql requests fail, then resort to tweet embed api
+    if (!testGraphql) {
+        syndication = true;
+        tweet = await requestSyndication(dispatcher, id);
+
+        const testSyndication = testResponse(tweet);
+
+        // if even syndication request failed, then cry out loud
+        if (!testSyndication) {
+            return { error: "fetch.fail" };
+        }
+    }
+
+    tweet = await tweet.json();
+
+    let media =
+        syndication
+            ? tweet.mediaDetails
+            : await extractGraphqlMedia(tweet, dispatcher, id, guestToken, cookie);
+
+    if (!media) return { error: "fetch.empty" };
 
     // check if there's a video at given index (/video/<index>)
     if (index >= 0 && index < media?.length) {
@@ -163,7 +250,7 @@ export default async function({ id, index, toGif, dispatcher, alwaysProxy }) {
         service: "twitter",
         type: "proxy",
         url, filename,
-    })
+    });
 
     switch (media?.length) {
         case undefined:

api/src/processing/services/xiaohongshu.js

@@ -0,0 +1,109 @@
+import { resolveRedirectingURL } from "../url.js";
+import { genericUserAgent } from "../../config.js";
+import { createStream } from "../../stream/manage.js";
+
+const https = (url) => {
+    return url.replace(/^http:/i, 'https:');
+}
+
+export default async function ({ id, token, shareId, h265, isAudioOnly, dispatcher }) {
+    let noteId = id;
+    let xsecToken = token;
+
+    if (!noteId) {
+        const patternMatch = await resolveRedirectingURL(
+            `https://xhslink.com/a/${shareId}`,
+            dispatcher
+        );
+
+        noteId = patternMatch?.id;
+        xsecToken = patternMatch?.token;
+    }
+
+    if (!noteId || !xsecToken) return { error: "fetch.short_link" };
+
+    const res = await fetch(`https://www.xiaohongshu.com/explore/${noteId}?xsec_token=${xsecToken}`, {
+        headers: {
+            "user-agent": genericUserAgent,
+        },
+        dispatcher,
+    });
+
+    const html = await res.text();
+
+    let note;
+    try {
+        const initialState = html
+            .split('<script>window.__INITIAL_STATE__=')[1]
+            .split('</script>')[0]
+            .replace(/:\s*undefined/g, ":null");
+
+        const data = JSON.parse(initialState);
+
+        const noteInfo = data?.note?.noteDetailMap;
+        if (!noteInfo) throw "no note detail map";
+
+        const currentNote = noteInfo[noteId];
+        if (!currentNote) throw "no current note in detail map";
+
+        note = currentNote.note;
+    } catch {}
+
+    if (!note) return { error: "fetch.empty" };
+
+    const video = note.video;
+    const images = note.imageList;
+
+    const filenameBase = `xiaohongshu_${noteId}`;
+
+    if (video) {
+        const videoFilename = `${filenameBase}.mp4`;
+        const audioFilename = `${filenameBase}_audio`;
+
+        let videoURL;
+
+        if (h265 && !isAudioOnly && video.consumer?.originVideoKey) {
+            videoURL = `https://sns-video-bd.xhscdn.com/${video.consumer.originVideoKey}`;
+        } else {
+            const h264Streams = video.media?.stream?.h264;
+
+            if (h264Streams?.length) {
+                videoURL = h264Streams.reduce((a, b) => Number(a?.videoBitrate) > Number(b?.videoBitrate) ? a : b).masterUrl;
+            }
+        }
+
+        if (!videoURL) return { error: "fetch.empty" };
+
+        return {
+            urls: https(videoURL),
+            filename: videoFilename,
+            audioFilename: audioFilename,
+        }
+    }
+
+    if (!images || images.length === 0) {
+        return { error: "fetch.empty" };
+    }
+
+    if (images.length === 1) {
+        return {
+            isPhoto: true,
+            urls: https(images[0].urlDefault),
+            filename: `${filenameBase}.jpg`,
+        }
+    }
+
+    const picker = images.map((image, i) => {
+        return {
+            type: "photo",
+            url: createStream({
+                service: "xiaohongshu",
+                type: "proxy",
+                url: https(image.urlDefault),
+                filename: `${filenameBase}_${i + 1}.jpg`,
+            })
+        }
+    });
+
+    return { picker };
+}

api/src/processing/services/youtube.js

@@ -4,7 +4,8 @@ import { fetch } from "undici";
 import { Innertube, Session } from "youtubei.js";
 
 import { env } from "../../config.js";
-import { getCookie, updateCookieValues } from "../cookie/manager.js";
+import { getCookie } from "../cookie/manager.js";
+import { getYouTubeSession } from "../helpers/youtube-session.js";
 
 const PLAYER_REFRESH_PERIOD = 1000 * 60 * 15; // ms
 
@@ -41,119 +42,110 @@ const hlsCodecList = {
     }
 }
 
+const clientsWithNoCipher = ['IOS', 'ANDROID', 'YTSTUDIO_ANDROID', 'YTMUSIC_ANDROID'];
+
 const videoQualities = [144, 240, 360, 480, 720, 1080, 1440, 2160, 4320];
 
-const transformSessionData = (cookie) => {
+const cloneInnertube = async (customFetch, useSession) => {
-    if (!cookie)
-        return;
-
-    const values = { ...cookie.values() };
-    const REQUIRED_VALUES = [ 'access_token', 'refresh_token' ];
-
-    if (REQUIRED_VALUES.some(x => typeof values[x] !== 'string')) {
-        return;
-    }
-
-    if (values.expires) {
-        values.expiry_date = values.expires;
-        delete values.expires;
-    } else if (!values.expiry_date) {
-        return;
-    }
-
-    return values;
-}
-
-const cloneInnertube = async (customFetch) => {
     const shouldRefreshPlayer = lastRefreshedAt + PLAYER_REFRESH_PERIOD < new Date();
+
+    const rawCookie = getCookie('youtube');
+    const cookie = rawCookie?.toString();
+
+    const sessionTokens = getYouTubeSession();
+    const retrieve_player = Boolean(sessionTokens || cookie);
+
+    if (useSession && env.ytSessionServer && !sessionTokens?.potoken) {
+        throw "no_session_tokens";
+    }
+
     if (!innertube || shouldRefreshPlayer) {
         innertube = await Innertube.create({
-            fetch: customFetch
+            fetch: customFetch,
+            retrieve_player,
+            cookie,
+            po_token: useSession ? sessionTokens?.potoken : undefined,
+            visitor_data: useSession ? sessionTokens?.visitor_data : undefined,
         });
         lastRefreshedAt = +new Date();
     }
 
     const session = new Session(
         innertube.session.context,
-        innertube.session.key,
+        innertube.session.api_key,
         innertube.session.api_version,
         innertube.session.account_index,
+        innertube.session.config_data,
         innertube.session.player,
-        undefined,
+        cookie,
         customFetch ?? innertube.session.http.fetch,
-        innertube.session.cache
+        innertube.session.cache,
+        sessionTokens?.potoken
     );
 
-    const cookie = getCookie('youtube_oauth');
-    const oauthData = transformSessionData(cookie);
-
-    if (!session.logged_in && oauthData) {
-        const tokensMod = {
-            ...oauthData,  // 复制 oauthData 中的所有属性
-            client: {
-                client_id: oauthData.client_id,
-                client_secret: oauthData.client_secret
-            }
-        };
-        delete tokensMod.client_id;
-        delete tokensMod.client_secret;
-
-        await session.oauth.init(tokensMod);
-        session.logged_in = true;
-    }
-
-    if (session.logged_in) {
-        if (session.oauth.shouldRefreshToken()) {
-            await session.oauth.refreshAccessToken();
-        }
-
-        const cookieValues = cookie.values();
-        const oldExpiry = new Date(cookieValues.expiry_date);
-        const newExpiry = new Date(session.oauth.oauth2_tokens.expiry_date);
-
-        if (oldExpiry.getTime() !== newExpiry.getTime()) {
-            updateCookieValues(cookie, {
-                ...session.oauth.client_id,
-                ...session.oauth.oauth2_tokens,
-                expiry_date: newExpiry.toISOString()
-            });
-        }
-    }
-
     const yt = new Innertube(session);
     return yt;
 }
 
-export default async function(o) {
+export default async function (o) {
+    const quality = o.quality === "max" ? 9000 : Number(o.quality);
+
+    let useHLS = o.youtubeHLS;
+    let innertubeClient = o.innertubeClient || env.customInnertubeClient || "IOS";
+
+    // HLS playlists from the iOS client don't contain the av1 video format.
+    if (useHLS && o.format === "av1") {
+        useHLS = false;
+    }
+
+    if (useHLS) {
+        innertubeClient = "IOS";
+    }
+
+    // iOS client doesn't have adaptive formats of resolution >1080p,
+    // so we use the WEB_EMBEDDED client instead for those cases
+    const useSession =
+        env.ytSessionServer && (
+            (
+                !useHLS
+                && innertubeClient === "IOS"
+                && (
+                    (quality > 1080 && o.format !== "h264")
+                    || (quality > 1080 && o.format !== "vp9")
+                )
+            )
+        );
+
+    if (useSession) {
+        innertubeClient = env.ytSessionInnertubeClient || "WEB_EMBEDDED";
+    }
+
     let yt;
     try {
         yt = await cloneInnertube(
             (input, init) => fetch(input, {
                 ...init,
                 dispatcher: o.dispatcher
-            })
+            }),
+            useSession
         );
     } catch (e) {
-        if (e.message?.endsWith("decipher algorithm")) {
+        if (e === "no_session_tokens") {
+            return { error: "youtube.no_session_tokens" };
+        } else if (e.message?.endsWith("decipher algorithm")) {
             return { error: "youtube.decipher" }
         } else if (e.message?.includes("refresh access token")) {
             return { error: "youtube.token_expired" }
         } else throw e;
     }
 
-    let useHLS = o.youtubeHLS;
-
-    // HLS playlists don't contain the av1 video format, at least with the iOS client
-    if (useHLS && o.format === "av1") {
-        useHLS = false;
-    }
-
     let info;
     try {
-        info = await yt.getBasicInfo(o.id, useHLS ? 'IOS' : 'WEB');
+        info = await yt.getBasicInfo(o.id, innertubeClient);
     } catch (e) {
         if (e?.info) {
-            const errorInfo = JSON.parse(e?.info);
+            let errorInfo;
+            try { errorInfo = JSON.parse(e?.info); } catch {}
 
             if (errorInfo?.reason === "This video is private") {
                 return { error: "content.video.private" };
@@ -175,12 +167,12 @@ export default async function(o) {
     const playability = info.playability_status;
     const basicInfo = info.basic_info;
 
-    switch(playability.status) {
+    switch (playability.status) {
         case "LOGIN_REQUIRED":
             if (playability.reason.endsWith("bot")) {
                 return { error: "youtube.login" }
             }
-            if (playability.reason.endsWith("age")) {
+            if (playability.reason.endsWith("age") || playability.reason.endsWith("inappropriate for some users.")) {
                 return { error: "content.video.age" }
             }
             if (playability?.error_screen?.reason?.text === "Private video") {
@@ -225,15 +217,13 @@ export default async function(o) {
         }
     }
 
-    const quality = o.quality === "max" ? 9000 : Number(o.quality);
-
     const normalizeQuality = res => {
-        const shortestSide = res.height > res.width ? res.width : res.height;
+        const shortestSide = Math.min(res.height, res.width);
         return videoQualities.find(qual => qual >= shortestSide);
     }
 
     let video, audio, dubbedLanguage,
-        codec = o.format || "h264";
+        codec = o.format || "h264", itag = o.itag;
 
     if (useHLS) {
         const hlsManifest = info.streaming_data.hls_manifest_url;
@@ -250,7 +240,7 @@ export default async function(o) {
             } else {
                 throw new Error("couldn't fetch the HLS playlist");
             }
-        }).catch(() => {});
+        }).catch(() => { });
 
         if (!fetchedHlsManifest) {
             return { error: "youtube.no_hls_streams" };
@@ -339,17 +329,21 @@ export default async function(o) {
             Number(b.bitrate) - Number(a.bitrate)
         ).forEach(format => {
             Object.keys(codecList).forEach(yCodec => {
+                const matchingItag = slot => !itag?.[slot] || itag[slot] === format.itag;
                 const sorted = sorted_formats[yCodec];
                 const goodFormat = checkFormat(format, yCodec);
                 if (!goodFormat) return;
 
-                if (format.has_video) {
+                if (format.has_video && matchingItag('video')) {
                     sorted.video.push(format);
-                    if (!sorted.bestVideo) sorted.bestVideo = format;
+                    if (!sorted.bestVideo)
+                        sorted.bestVideo = format;
                 }
-                if (format.has_audio) {
+
+                if (format.has_audio && matchingItag('audio')) {
                     sorted.audio.push(format);
-                    if (!sorted.bestAudio) sorted.bestAudio = format;
+                    if (!sorted.bestAudio)
+                        sorted.bestAudio = format;
                 }
             })
         });
@@ -411,6 +405,10 @@ export default async function(o) {
         }
     }
 
+    if (video?.drm_families || audio?.drm_families) {
+        return { error: "youtube.drm" };
+    }
+
     const fileMetadata = {
         title: basicInfo.title.trim(),
         artist: basicInfo.author.replace("- Topic", "").trim()
@@ -436,6 +434,18 @@ export default async function(o) {
         youtubeDubName: dubbedLanguage || false,
     }
 
+    itag = {
+        video: video?.itag,
+        audio: audio?.itag
+    };
+
+    const originalRequest = {
+        ...o,
+        dispatcher: undefined,
+        itag,
+        innertubeClient
+    };
+
     if (audio && o.isAudioOnly) {
         let bestAudio = codec === "h264" ? "m4a" : "opus";
         let urls = audio.url;
@@ -445,6 +455,10 @@ export default async function(o) {
             urls = audio.uri;
         }
 
+        if (!clientsWithNoCipher.includes(innertubeClient) && innertube) {
+            urls = audio.decipher(innertube.session.player);
+        }
+
         return {
             type: "audio",
             isAudioOnly: true,
@@ -453,6 +467,7 @@ export default async function(o) {
             fileMetadata,
             bestAudio,
             isHLS: useHLS,
+            originalRequest
         }
     }
 
@@ -471,12 +486,18 @@ export default async function(o) {
                 width: video.width,
                 height: video.height,
             });
+
             filenameAttributes.resolution = `${video.width}x${video.height}`;
             filenameAttributes.extension = codecList[codec].container;
 
+            if (!clientsWithNoCipher.includes(innertubeClient) && innertube) {
+                video = video.decipher(innertube.session.player);
+                audio = audio.decipher(innertube.session.player);
+            } else {
                 video = video.url;
                 audio = audio.url;
             }
+        }
 
         filenameAttributes.qualityLabel = `${resolution}p`;
         filenameAttributes.youtubeFormat = codec;
@@ -490,6 +511,7 @@ export default async function(o) {
             filenameAttributes,
             fileMetadata,
             isHLS: useHLS,
+            originalRequest
         }
     }
 

api/src/processing/url.js

@@ -3,6 +3,7 @@ import { strict as assert } from "node:assert";
 
 import { env } from "../config.js";
 import { services } from "./service-config.js";
+import { getRedirectingURL } from "../misc/utils.js";
 import { friendlyServiceName } from "./service-alias.js";
 
 function aliasURL(url) {
@@ -92,9 +93,30 @@ function aliasURL(url) {
                 url.hostname = 'vk.com';
             }
             break;
+
+        case "xhslink":
+            if (url.hostname === 'xhslink.com' && parts.length === 3) {
+                url = new URL(`https://www.xiaohongshu.com/a/${parts[2]}`);
+            }
+            break;
+
+        case "loom":
+            const idPart = parts[parts.length - 1];
+            if (idPart.length > 32) {
+                url.pathname = `/share/${idPart.slice(-32)}`;
+            }
+            break;
+            
+        case "redd":
+            /* reddit short video links can be treated by changing https://v.redd.it/<id>
+            to https://reddit.com/video/<id>.*/
+            if (url.hostname === "v.redd.it" && parts.length === 2) {
+                url = new URL(`https://www.reddit.com/video/${parts[1]}`);
+            }
+            break;
     }
 
-    return url
+    return url;
 }
 
 function cleanURL(url) {
@@ -114,36 +136,41 @@ function cleanURL(url) {
             break;
         case "vk":
             if (url.pathname.includes('/clip') && url.searchParams.get('z')) {
-                limitQuery('z')
+                limitQuery('z');
             }
             break;
         case "youtube":
             if (url.searchParams.get('v')) {
-                limitQuery('v')
+                limitQuery('v');
             }
             break;
         case "rutube":
             if (url.searchParams.get('p')) {
-                limitQuery('p')
+                limitQuery('p');
             }
             break;
         case "twitter":
             if (url.searchParams.get('post_id')) {
-                limitQuery('post_id')
+                limitQuery('post_id');
+            }
+            break;
+        case "xiaohongshu":
+            if (url.searchParams.get('xsec_token')) {
+                limitQuery('xsec_token');
             }
             break;
     }
 
     if (stripQuery) {
-        url.search = ''
+        url.search = '';
     }
 
-    url.username = url.password = url.port = url.hash = ''
+    url.username = url.password = url.port = url.hash = '';
 
     if (url.pathname.endsWith('/'))
         url.pathname = url.pathname.slice(0, -1);
 
-    return url
+    return url;
 }
 
 function getHostIfValid(url) {
@@ -181,6 +208,11 @@ export function extract(url) {
     }
 
     if (!env.enabledServices.has(host)) {
+        // show a different message when youtube is disabled on official instances
+        // as it only happens when shit hits the fan
+        if (new URL(env.apiURL).hostname.endsWith(".imput.net") && host === "youtube") {
+            return { error: "youtube.temporary_disabled" };
+        }
         return { error: "service.disabled" };
     }
 
@@ -206,3 +238,17 @@ export function extract(url) {
 
     return { host, patternMatch };
 }
+
+export async function resolveRedirectingURL(url, dispatcher, headers) {
+    const originalService = getHostIfValid(normalizeURL(url));
+    if (!originalService) return;
+
+    const canonicalURL = await getRedirectingURL(url, dispatcher, headers);
+    if (!canonicalURL) return;
+
+    const { host, patternMatch } = extract(normalizeURL(canonicalURL));
+
+    if (host === originalService) {
+        return patternMatch;
+    }
+}

api/src/security/api-keys.js

@@ -1,8 +1,8 @@
 import { env } from "../config.js";
-import { readFile } from "node:fs/promises";
 import { Green, Yellow } from "../misc/console-text.js";
 import ip from "ipaddr.js";
 import * as cluster from "../misc/cluster.js";
+import { FileWatcher } from "../misc/file-watcher.js";
 
 // this function is a modified variation of code
 // from https://stackoverflow.com/a/32402438/14855621
@@ -13,7 +13,7 @@ const generateWildcardRegex = rule => {
 
 const UUID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
 
-let keys = {};
+let keys = {}, reader = null;
 
 const ALLOWED_KEYS = new Set(['name', 'ips', 'userAgents', 'limit']);
 
@@ -118,34 +118,39 @@ const formatKeys = (keyData) => {
 }
 
 const updateKeys = (newKeys) => {
+    validateKeys(newKeys);
+
+    cluster.broadcast({ api_keys: newKeys });
+
     keys = formatKeys(newKeys);
 }
 
-const loadKeys = async (source) => {
+const loadRemoteKeys = async (source) => {
-    let updated;
+    updateKeys(
-    if (source.protocol === 'file:') {
+        await fetch(source).then(a => a.json())
-        const pathname = source.pathname === '/' ? '' : source.pathname;
-        updated = JSON.parse(
-            await readFile(
-                decodeURIComponent(source.host + pathname),
-                'utf8'
-            )
     );
-    } else {
+}
-        updated = await fetch(source).then(a => a.json());
-    }
 
-    validateKeys(updated);
+const loadLocalKeys = async () => {
-
+    updateKeys(
-    cluster.broadcast({ api_keys: updated });
+        JSON.parse(await reader.read())
-
+    );
-    updateKeys(updated);
 }
 
 const wrapLoad = (url, initial = false) => {
-    loadKeys(url)
+    let load = loadRemoteKeys.bind(null, url);
-    .then(() => {
+
+    if (url.protocol === 'file:') {
         if (initial) {
+            reader = FileWatcher.fromFileProtocol(url);
+            reader.on('file-updated', () => wrapLoad(url));
+        }
+
+        load = loadLocalKeys;
+    }
+
+    load().then(() => {
+        if (initial || reader) {
             console.log(`${Green('[✓]')} api keys loaded successfully!`)
         }
     })
@@ -214,7 +219,7 @@ export const validateAuthorization = (req) => {
 export const setup = (url) => {
     if (cluster.isPrimary) {
         wrapLoad(url, true);
-        if (env.keyReloadInterval > 0) {
+        if (env.keyReloadInterval > 0 && url.protocol !== 'file:') {
             setInterval(() => wrapLoad(url), env.keyReloadInterval * 1000);
         }
     } else if (cluster.isWorker) {

api/src/security/jwt.js

@@ -6,12 +6,19 @@ import { env } from "../config.js";
 const toBase64URL = (b) => Buffer.from(b).toString("base64url");
 const fromBase64URL = (b) => Buffer.from(b, "base64url").toString();
 
-const makeHmac = (header, payload) =>
+const makeHmac = (data) => {
-    createHmac("sha256", env.jwtSecret)
+    return createHmac("sha256", env.jwtSecret)
-        .update(`${header}.${payload}`)
+            .update(data)
             .digest("base64url");
+}
 
-const generate = () => {
+const sign = (header, payload) =>
+        makeHmac(`${header}.${payload}`);
+
+const getIPHash = (ip) =>
+        makeHmac(ip).slice(0, 8);
+
+const generate = (ip) => {
     const exp = Math.floor(new Date().getTime() / 1000) + env.jwtLifetime;
 
     const header = toBase64URL(JSON.stringify({
@@ -21,10 +28,11 @@ const generate = () => {
 
     const payload = toBase64URL(JSON.stringify({
         jti: nanoid(8),
+        sub: getIPHash(ip),
         exp,
     }));
 
-    const signature = makeHmac(header, payload);
+    const signature = sign(header, payload);
 
     return {
         token: `${header}.${payload}.${signature}`,
@@ -32,7 +40,7 @@ const generate = () => {
     };
 }
 
-const verify = (jwt) => {
+const verify = (jwt, ip) => {
     const [header, payload, signature] = jwt.split(".", 3);
     const timestamp = Math.floor(new Date().getTime() / 1000);
 
@@ -40,17 +48,16 @@ const verify = (jwt) => {
         return false;
     }
 
-    const verifySignature = makeHmac(header, payload);
+    const verifySignature = sign(header, payload);
 
     if (verifySignature !== signature) {
         return false;
     }
 
-    if (timestamp >= JSON.parse(fromBase64URL(payload)).exp) {
+    const data = JSON.parse(fromBase64URL(payload));
-        return false;
-    }
 
-    return true;
+    return getIPHash(ip) === data.sub
+            && timestamp <= data.exp;
 }
 
 export default {

api/src/stream/internal-hls.js

@@ -1,5 +1,6 @@
 import HLS from "hls-parser";
 import { createInternalStream } from "./manage.js";
+import { request } from "undici";
 
 function getURL(url) {
     try {
@@ -16,16 +17,18 @@ function transformObject(streamInfo, hlsObject) {
 
     let fullUrl;
     if (getURL(hlsObject.uri)) {
-        fullUrl = hlsObject.uri;
+        fullUrl = new URL(hlsObject.uri);
     } else {
         fullUrl = new URL(hlsObject.uri, streamInfo.url);
     }
 
+    if (fullUrl.hostname !== '127.0.0.1') {
         hlsObject.uri = createInternalStream(fullUrl.toString(), streamInfo);
 
         if (hlsObject.map) {
             hlsObject.map = transformObject(streamInfo, hlsObject.map);
         }
+    }
 
     return hlsObject;
 }
@@ -53,8 +56,11 @@ function transformMediaPlaylist(streamInfo, hlsPlaylist) {
 
 const HLS_MIME_TYPES = ["application/vnd.apple.mpegurl", "audio/mpegurl", "application/x-mpegURL"];
 
-export function isHlsResponse (req) {
+export function isHlsResponse(req, streamInfo) {
-    return HLS_MIME_TYPES.includes(req.headers['content-type']);
+    return HLS_MIME_TYPES.includes(req.headers['content-type'])
+        // bluesky's cdn responds with wrong content-type for the hls playlist,
+        // so we enforce it here until they fix it
+        || (streamInfo.service === 'bsky' && streamInfo.url.endsWith('.m3u8'));
 }
 
 export async function handleHlsPlaylist(streamInfo, req, res) {
@@ -69,3 +75,67 @@ export async function handleHlsPlaylist(streamInfo, req, res) {
 
     res.send(hlsPlaylist);
 }
+
+async function getSegmentSize(url, config) {
+    const segmentResponse = await request(url, {
+        ...config,
+        throwOnError: true
+    });
+
+    if (segmentResponse.headers['content-length']) {
+        segmentResponse.body.dump();
+        return +segmentResponse.headers['content-length'];
+    }
+
+    // if the response does not have a content-length
+    // header, we have to compute it ourselves
+    let size = 0;
+
+    for await (const data of segmentResponse.body) {
+        size += data.length;
+    }
+
+    return size;
+}
+
+export async function probeInternalHLSTunnel(streamInfo) {
+    const { url, headers, dispatcher, signal } = streamInfo;
+
+    // remove all falsy headers
+    Object.keys(headers).forEach(key => {
+        if (!headers[key]) delete headers[key];
+    });
+
+    const config = { headers, dispatcher, signal, maxRedirections: 16 };
+
+    const manifestResponse = await fetch(url, config);
+
+    const manifest = HLS.parse(await manifestResponse.text());
+    if (manifest.segments.length === 0)
+        return -1;
+
+    const segmentSamples = await Promise.all(
+        Array(5).fill().map(async () => {
+            const manifestIdx = Math.floor(Math.random() * manifest.segments.length);
+            const randomSegment = manifest.segments[manifestIdx];
+            if (!randomSegment.uri)
+                throw "segment is missing URI";
+
+            let segmentUrl;
+
+            if (getURL(randomSegment.uri)) {
+                segmentUrl = new URL(randomSegment.uri);
+            } else {
+                segmentUrl = new URL(randomSegment.uri, streamInfo.url);
+            }
+
+            const segmentSize = await getSegmentSize(segmentUrl, config) / randomSegment.duration;
+            return segmentSize;
+        })
+    );
+
+    const averageBitrate = segmentSamples.reduce((a, b) => a + b) / segmentSamples.length;
+    const totalDuration = manifest.segments.reduce((acc, segment) => acc + segment.duration, 0);
+
+    return averageBitrate * totalDuration;
+}

api/src/stream/internal.js

@@ -1,13 +1,13 @@
 import { request } from "undici";
 import { Readable } from "node:stream";
 import { closeRequest, getHeaders, pipe } from "./shared.js";
-import { handleHlsPlaylist, isHlsResponse } from "./internal-hls.js";
+import { handleHlsPlaylist, isHlsResponse, probeInternalHLSTunnel } from "./internal-hls.js";
 
 const CHUNK_SIZE = BigInt(8e6); // 8 MB
 const min = (a, b) => a < b ? a : b;
 
 async function* readChunks(streamInfo, size) {
-    let read = 0n;
+    let read = 0n, chunksSinceTransplant = 0;
     while (read < size) {
         if (streamInfo.controller.signal.aborted) {
             throw new Error("controller aborted");
@@ -19,9 +19,20 @@ async function* readChunks(streamInfo, size) {
                 Range: `bytes=${read}-${read + CHUNK_SIZE}`
             },
             dispatcher: streamInfo.dispatcher,
-            signal: streamInfo.controller.signal
+            signal: streamInfo.controller.signal,
+            maxRedirections: 4
         });
 
+        if (chunk.statusCode === 403 && chunksSinceTransplant >= 3 && streamInfo.transplant) {
+            chunksSinceTransplant = 0;
+            try {
+                await streamInfo.transplant(streamInfo.dispatcher);
+                continue;
+            } catch {}
+        }
+
+        chunksSinceTransplant++;
+
         const expected = min(CHUNK_SIZE, size - read);
         const received = BigInt(chunk.headers['content-length']);
 
@@ -42,7 +53,9 @@ async function handleYoutubeStream(streamInfo, res) {
     const cleanup = () => (res.end(), closeRequest(streamInfo.controller));
 
     try {
-        const req = await fetch(streamInfo.url, {
+        let req, attempts = 3;
+        while (attempts--) {
+            req = await fetch(streamInfo.url, {
                 headers: getHeaders('youtube'),
                 method: 'HEAD',
                 dispatcher: streamInfo.dispatcher,
@@ -50,9 +63,16 @@ async function handleYoutubeStream(streamInfo, res) {
             });
 
             streamInfo.url = req.url;
+            if (req.status === 403 && streamInfo.transplant) {
+                try {
+                    await streamInfo.transplant(streamInfo.dispatcher);
+                } catch {
+                    break;
+                }
+            } else break;
+        }
+
         const size = BigInt(req.headers.get('content-length'));
-        console.log("size=========>",size)
-        console.log("req.status=========>", req.status);
 
         if (req.status !== 200 || !size) {
             return cleanup();
@@ -98,10 +118,7 @@ async function handleGenericStream(streamInfo, res) {
         res.status(fileResponse.statusCode);
         fileResponse.body.on('error', () => {});
 
-        // bluesky's cdn responds with wrong content-type for the hls playlist,
+        const isHls = isHlsResponse(fileResponse, streamInfo);
-        // so we enforce it here until they fix it
-        const isHls = isHlsResponse(fileResponse)
-                      || (streamInfo.service === "bsky" && streamInfo.url.endsWith('.m3u8'));
 
         for (const [ name, value ] of Object.entries(fileResponse.headers)) {
             if (!isHls || name.toLowerCase() !== 'content-length') {
@@ -135,3 +152,40 @@ export function internalStream(streamInfo, res) {
 
     return handleGenericStream(streamInfo, res);
 }
+
+export async function probeInternalTunnel(streamInfo) {
+    try {
+        const signal = AbortSignal.timeout(3000);
+        const headers = {
+            ...Object.fromEntries(streamInfo.headers || []),
+            ...getHeaders(streamInfo.service),
+            host: undefined,
+            range: undefined
+        };
+
+        if (streamInfo.isHLS) {
+            return probeInternalHLSTunnel({
+                ...streamInfo,
+                signal,
+                headers
+            });
+        }
+
+        const response = await request(streamInfo.url, {
+            method: 'HEAD',
+            headers,
+            dispatcher: streamInfo.dispatcher,
+            signal,
+            maxRedirections: 16
+        });
+
+        if (response.statusCode !== 200)
+            throw "status is not 200 OK";
+
+        const size = +response.headers['content-length'];
+        if (isNaN(size))
+            throw "content-length is not a number";
+
+        return size;
+    } catch {}
+}

api/src/stream/manage.js

@@ -9,6 +9,7 @@ import { env } from "../config.js";
 import { closeRequest } from "./shared.js";
 import { decryptStream, encryptStream } from "../misc/crypto.js";
 import { hashHmac } from "../security/secrets.js";
+import { zip } from "../misc/utils.js";
 
 // optional dependency
 const freebind = env.freebindCIDR && await import('freebind').catch(() => {});
@@ -40,6 +41,7 @@ export function createStream(obj) {
             audioFormat: obj.audioFormat,
 
             isHLS: obj.isHLS || false,
+            originalRequest: obj.originalRequest
         };
 
     // FIXME: this is now a Promise, but it is not awaited
@@ -68,10 +70,47 @@ export function createStream(obj) {
     return streamLink.toString();
 }
 
-export function getInternalStream(id) {
+export function createProxyTunnels(info) {
+    const proxyTunnels = [];
+
+    let urls = info.url;
+
+    if (typeof urls === "string") {
+        urls = [urls];
+    }
+
+    for (const url of urls) {
+        proxyTunnels.push(
+            createStream({
+                url,
+                type: "proxy",
+
+                service: info?.service,
+                headers: info?.headers,
+                requestIP: info?.requestIP,
+
+                originalRequest: info?.originalRequest
+            })
+        );
+    }
+
+    return proxyTunnels;
+}
+
+export function getInternalTunnel(id) {
     return internalStreamCache.get(id);
 }
 
+export function getInternalTunnelFromURL(url) {
+    url = new URL(url);
+    if (url.hostname !== '127.0.0.1') {
+        return;
+    }
+
+    const id = url.searchParams.get('id');
+    return getInternalTunnel(id);
+}
+
 export function createInternalStream(url, obj = {}) {
     assert(typeof url === 'string');
 
@@ -100,6 +139,7 @@ export function createInternalStream(url, obj = {}) {
         controller,
         dispatcher,
         isHLS: obj.isHLS,
+        transplant: obj.transplant
     });
 
     let streamLink = new URL('/itunnel', `http://127.0.0.1:${env.tunnelPort}`);
@@ -115,23 +155,86 @@ export function createInternalStream(url, obj = {}) {
     return streamLink.toString();
 }
 
-export function destroyInternalStream(url) {
+function getInternalTunnelId(url) {
     url = new URL(url);
     if (url.hostname !== '127.0.0.1') {
         return;
     }
 
-    const id = url.searchParams.get('id');
+    return url.searchParams.get('id');
+}
+
+export function destroyInternalStream(url) {
+    const id = getInternalTunnelId(url);
 
     if (internalStreamCache.has(id)) {
-        closeRequest(getInternalStream(id)?.controller);
+        closeRequest(getInternalTunnel(id)?.controller);
         internalStreamCache.delete(id);
     }
 }
 
+const transplantInternalTunnels = function(tunnelUrls, transplantUrls) {
+    if (tunnelUrls.length !== transplantUrls.length) {
+        return;
+    }
+
+    for (const [ tun, url ] of zip(tunnelUrls, transplantUrls)) {
+        const id = getInternalTunnelId(tun);
+        const itunnel = getInternalTunnel(id);
+
+        if (!itunnel) continue;
+        itunnel.url = url;
+    }
+}
+
+const transplantTunnel = async function (dispatcher) {
+    if (this.pendingTransplant) {
+        await this.pendingTransplant;
+        return;
+    }
+
+    let finished;
+    this.pendingTransplant = new Promise(r => finished = r);
+
+    try {
+        const handler = await import(`../processing/services/${this.service}.js`);
+        const response = await handler.default({
+            ...this.originalRequest,
+            dispatcher
+        });
+
+        if (!response.urls) {
+            return;
+        }
+
+        response.urls = [response.urls].flat();
+        if (this.originalRequest.isAudioOnly && response.urls.length > 1) {
+            response.urls = [response.urls[1]];
+        } else if (this.originalRequest.isAudioMuted) {
+            response.urls = [response.urls[0]];
+        }
+
+        const tunnels = [this.urls].flat();
+        if (tunnels.length !== response.urls.length) {
+            return;
+        }
+
+        transplantInternalTunnels(tunnels, response.urls);
+    }
+    catch {}
+    finally {
+        finished();
+        delete this.pendingTransplant;
+    }
+}
+
 function wrapStream(streamInfo) {
     const url = streamInfo.urls;
 
+    if (streamInfo.originalRequest) {
+        streamInfo.transplant = transplantTunnel.bind(streamInfo);
+    }
+
     if (typeof url === 'string') {
         streamInfo.urls = createInternalStream(url, streamInfo);
     } else if (Array.isArray(url)) {

api/src/stream/shared.js

@@ -1,5 +1,7 @@
 import { genericUserAgent } from "../config.js";
 import { vkClientAgent } from "../processing/services/vk.js";
+import { getInternalTunnelFromURL } from "./manage.js";
+import { probeInternalTunnel } from "./internal.js";
 
 const defaultHeaders = {
     'user-agent': genericUserAgent
@@ -47,3 +49,40 @@ export function pipe(from, to, done) {
 
     from.pipe(to);
 }
+
+export async function estimateTunnelLength(streamInfo, multiplier = 1.1) {
+    let urls = streamInfo.urls;
+    if (!Array.isArray(urls)) {
+        urls = [ urls ];
+    }
+
+    const internalTunnels = urls.map(getInternalTunnelFromURL);
+    if (internalTunnels.some(t => !t))
+        return -1;
+
+    const sizes = await Promise.all(internalTunnels.map(probeInternalTunnel));
+    const estimatedSize = sizes.reduce(
+        // if one of the sizes is missing, let's just make a very
+        // bold guess that it's the same size as the existing one
+        (acc, cur) => cur <= 0 ? acc * 2 : acc + cur,
+        0
+    );
+
+    if (isNaN(estimatedSize) || estimatedSize <= 0) {
+        return -1;
+    }
+
+    return Math.floor(estimatedSize * multiplier);
+}
+
+export function estimateAudioMultiplier(streamInfo) {
+    if (streamInfo.audioFormat === 'wav') {
+        return 1411 / 128;
+    }
+
+    if (streamInfo.audioCopy) {
+        return 1;
+    }
+
+    return streamInfo.audioBitrate / 128;
+}

api/src/stream/stream.js

@@ -10,20 +10,20 @@ export default async function(res, streamInfo) {
                 return await stream.proxy(streamInfo, res);
 
             case "internal":
-                return internalStream(streamInfo, res);
+                return await internalStream(streamInfo.data, res);
 
             case "merge":
-                return stream.merge(streamInfo, res);
+                return await stream.merge(streamInfo, res);
 
             case "remux":
             case "mute":
-                return stream.remux(streamInfo, res);
+                return await stream.remux(streamInfo, res);
 
             case "audio":
-                return stream.convertAudio(streamInfo, res);
+                return await stream.convertAudio(streamInfo, res);
 
             case "gif":
-                return stream.convertGif(streamInfo, res);
+                return await stream.convertGif(streamInfo, res);
         }
 
         closeResponse(res);

api/src/stream/types.js

@@ -1,4 +1,4 @@
-import { request } from "undici";
+import { Agent, request } from "undici";
 import ffmpeg from "ffmpeg-static";
 import { spawn } from "child_process";
 import { create as contentDisposition } from "content-disposition-header";
@@ -6,7 +6,7 @@ import { create as contentDisposition } from "content-disposition-header";
 import { env } from "../config.js";
 import { destroyInternalStream } from "./manage.js";
 import { hlsExceptions } from "../processing/service-config.js";
-import { getHeaders, closeRequest, closeResponse, pipe } from "./shared.js";
+import { getHeaders, closeRequest, closeResponse, pipe, estimateTunnelLength, estimateAudioMultiplier } from "./shared.js";
 
 const ffmpegArgs = {
     webm: ["-c:v", "copy", "-c:a", "copy"],
@@ -29,7 +29,7 @@ const convertMetadataToFFmpeg = (metadata) => {
 
     for (const [ name, value ] of Object.entries(metadata)) {
         if (metadataTags.includes(name)) {
-            args.push('-metadata', `${name}=${value.replace(/[\u0000-\u0009]/g, "")}`);
+            args.push('-metadata', `${name}=${value.replace(/[\u0000-\u0009]/g, "")}`); // skipcq: JS-0004
         } else {
             throw `${name} metadata tag is not supported.`;
         }
@@ -60,6 +60,8 @@ const getCommand = (args) => {
     return [ffmpeg, args]
 }
 
+const defaultAgent = new Agent();
+
 const proxy = async (streamInfo, res) => {
     const abortController = new AbortController();
     const shutdown = () => (
@@ -78,7 +80,8 @@ const proxy = async (streamInfo, res) => {
                 Range: streamInfo.range
             },
             signal: abortController.signal,
-            maxRedirections: 16
+            maxRedirections: 16,
+            dispatcher: defaultAgent,
         });
 
         res.status(statusCode);
@@ -95,7 +98,7 @@ const proxy = async (streamInfo, res) => {
     }
 }
 
-const merge = (streamInfo, res) => {
+const merge = async (streamInfo, res) => {
     let process;
     const shutdown = () => (
         killProcess(process),
@@ -109,7 +112,7 @@ const merge = (streamInfo, res) => {
     try {
         if (streamInfo.urls.length !== 2) return shutdown();
 
-        const format = streamInfo.filename.split('.')[streamInfo.filename.split('.').length - 1];
+        const format = streamInfo.filename.split('.').pop();
 
         let args = [
             '-loglevel', '-8',
@@ -149,6 +152,7 @@ const merge = (streamInfo, res) => {
 
         res.setHeader('Connection', 'keep-alive');
         res.setHeader('Content-Disposition', contentDisposition(streamInfo.filename));
+        res.setHeader('Estimated-Content-Length', await estimateTunnelLength(streamInfo));
 
         pipe(muxOutput, res, shutdown);
 
@@ -159,7 +163,7 @@ const merge = (streamInfo, res) => {
     }
 }
 
-const remux = (streamInfo, res) => {
+const remux = async (streamInfo, res) => {
     let process;
     const shutdown = () => (
         killProcess(process),
@@ -193,7 +197,7 @@ const remux = (streamInfo, res) => {
             args.push('-bsf:a', 'aac_adtstoasc');
         }
 
-        let format = streamInfo.filename.split('.')[streamInfo.filename.split('.').length - 1];
+        let format = streamInfo.filename.split('.').pop();
         if (format === "mp4") {
             args.push('-movflags', 'faststart+frag_keyframe+empty_moov')
         }
@@ -212,6 +216,7 @@ const remux = (streamInfo, res) => {
 
         res.setHeader('Connection', 'keep-alive');
         res.setHeader('Content-Disposition', contentDisposition(streamInfo.filename));
+        res.setHeader('Estimated-Content-Length', await estimateTunnelLength(streamInfo));
 
         pipe(muxOutput, res, shutdown);
 
@@ -222,7 +227,7 @@ const remux = (streamInfo, res) => {
     }
 }
 
-const convertAudio = (streamInfo, res) => {
+const convertAudio = async (streamInfo, res) => {
     let process;
     const shutdown = () => (
         killProcess(process),
@@ -281,6 +286,13 @@ const convertAudio = (streamInfo, res) => {
 
         res.setHeader('Connection', 'keep-alive');
         res.setHeader('Content-Disposition', contentDisposition(streamInfo.filename));
+        res.setHeader(
+            'Estimated-Content-Length',
+            await estimateTunnelLength(
+                streamInfo,
+                estimateAudioMultiplier(streamInfo) * 1.1
+            )
+        );
 
         pipe(muxOutput, res, shutdown);
         res.on('finish', shutdown);
@@ -289,7 +301,7 @@ const convertAudio = (streamInfo, res) => {
     }
 }
 
-const convertGif = (streamInfo, res) => {
+const convertGif = async (streamInfo, res) => {
     let process;
     const shutdown = () => (killProcess(process), closeResponse(res));
 
@@ -318,6 +330,7 @@ const convertGif = (streamInfo, res) => {
 
         res.setHeader('Connection', 'keep-alive');
         res.setHeader('Content-Disposition', contentDisposition(streamInfo.filename));
+        res.setHeader('Estimated-Content-Length', await estimateTunnelLength(streamInfo, 60));
 
         pipe(muxOutput, res, shutdown);
 

api/src/util/test.js

@@ -4,6 +4,7 @@ import { env } from "../config.js";
 import { runTest } from "../misc/run-test.js";
 import { loadJSON } from "../misc/load-from-fs.js";
 import { Red, Bright } from "../misc/console-text.js";
+import { setGlobalDispatcher, ProxyAgent } from "undici";
 import { randomizeCiphers } from "../misc/randomize-ciphers.js";
 
 import { services } from "../processing/service-config.js";
@@ -13,7 +14,11 @@ const getTests = (service) => loadJSON(getTestPath(service));
 
 // services that are known to frequently fail due to external
 // factors (e.g. rate limiting)
-const finnicky = new Set(['bilibili', 'instagram', 'facebook', 'youtube', 'vk', 'twitter']);
+const finnicky = new Set(
+    process.env.TEST_IGNORE_SERVICES
+    ? process.env.TEST_IGNORE_SERVICES.split(',')
+    : ['bilibili', 'instagram', 'facebook', 'youtube', 'vk', 'twitter', 'reddit']
+);
 
 const runTestsFor = async (service) => {
     const tests = getTests(service);
@@ -64,6 +69,14 @@ const printHeader = (service, padLen) => {
     console.log(service + '='.repeat(50));
 }
 
+if (env.externalProxy) {
+    setGlobalDispatcher(new ProxyAgent(env.externalProxy));
+}
+
+env.streamLifespan = 10000;
+env.apiURL = 'http://x/';
+randomizeCiphers();
+
 const action = process.argv[2];
 switch (action) {
     case "get-services":
@@ -86,9 +99,6 @@ switch (action) {
         break;
 
     case "run-tests-for":
-        env.streamLifespan = 10000;
-        env.apiURL = 'http://x/';
-        randomizeCiphers();
 
         try {
             const { softFails } = await runTestsFor(process.argv[3]);
@@ -104,10 +114,6 @@ switch (action) {
         const maxHeaderLen = Object.keys(services).reduce((n, v) => v.length > n ? v.length : n, 0);
         const failCounters = {};
 
-        env.streamLifespan = 10000;
-        env.apiURL = 'http://x/';
-        randomizeCiphers();
-
         for (const service in services) {
             printHeader(service, maxHeaderLen);
             const { fails, softFails } = await runTestsFor(service);

api/src/util/tests/bilibili.json

@@ -41,7 +41,7 @@
     },
     {
         "name": "b23.tv shortlink",
-        "url": "https://b23.tv/lbMyOI9",
+        "url": "https://b23.tv/av32430100",
         "params": {},
         "expected": {
             "code": 200,

api/src/util/tests/bsky.json

@@ -54,7 +54,25 @@
         "params": {},
         "expected": {
             "code": 200,
-            "status": "redirect"
+            "status": "tunnel"
+        }
+    },
+    {
+        "name": "gif with a quoted post",
+        "url": "https://bsky.app/profile/imlunahey.com/post/3lgajpn5dtk2t",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
+    },
+    {
+        "name": "gif alone in a post",
+        "url": "https://bsky.app/profile/imlunahey.com/post/3lgah3ovxnc2q",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
         }
     },
     {

api/src/util/tests/facebook.json

@@ -29,7 +29,6 @@
     {
         "name": "shortlink video",
         "url": "https://fb.watch/r1K6XHMfGT/",
-        "canFail": true,
         "params": {},
         "expected": {
             "code": 200,
@@ -39,7 +38,6 @@
     {
         "name": "reel video",
         "url": "https://web.facebook.com/reel/730293269054758",
-        "canFail": true,
         "params": {},
         "expected": {
             "code": 200,
@@ -48,7 +46,7 @@
     },
     {
         "name": "shared video link",
-        "url": "https://www.facebook.com/share/v/NEf87jbPTvFE8LsL/",
+        "url": "https://www.facebook.com/share/v/6EJK4Z8EAEAHtz8K/",
         "params": {},
         "expected": {
             "code": 200,

api/src/util/tests/instagram.json

@@ -1,11 +1,11 @@
 [
     {
         "name": "single photo post",
-        "url": "https://www.instagram.com/p/CwIgW8Yu5-I/",
+        "url": "https://www.instagram.com/p/DFx6KVduFWy/",
         "params": {},
         "expected": {
             "code": 200,
-            "status": "redirect"
+            "status": "tunnel"
         }
     },
     {
@@ -19,7 +19,7 @@
     },
     {
         "name": "reel",
-        "url": "https://www.instagram.com/reel/CoEBV3eM4QR/",
+        "url": "https://www.instagram.com/reel/DFQe23tOWKz/",
         "params": {},
         "expected": {
             "code": 200,
@@ -37,7 +37,7 @@
     },
     {
         "name": "reel (isAudioOnly)",
-        "url": "https://www.instagram.com/reel/CoEBV3eM4QR/",
+        "url": "https://www.instagram.com/reel/DFQe23tOWKz/",
         "params": {
             "downloadMode": "audio"
         },
@@ -48,7 +48,7 @@
     },
     {
         "name": "reel (isAudioMuted)",
-        "url": "https://www.instagram.com/reel/CoEBV3eM4QR/",
+        "url": "https://www.instagram.com/reel/DFQe23tOWKz/",
         "params": {
             "downloadMode": "mute"
         },
@@ -119,5 +119,16 @@
             "code": 200,
             "status": "redirect"
         }
+    },
+    {
+        "name": "private instagram post",
+        "url": "https://www.instagram.com/p/C5_A1TQNPrYw4c2g9KAUTPUl8RVHqiAdAcOOSY0",
+        "canFail": true,
+        "params": {},
+        "expected": {
+            "code": 400,
+            "status": "error",
+            "errorCode": "error.api.content.post.private"
+        }
     }
 ]

api/src/util/tests/loom.json

@@ -1,7 +1,7 @@
 [
     {
         "name": "1080p video",
-        "url": "https://www.loom.com/share/313bf71d20ca47b2a35b6634cefdb761",
+        "url": "https://www.loom.com/share/d165fd054a294d8a8587807bcc50c885",
         "params": {},
         "expected": {
             "code": 200,
@@ -10,7 +10,7 @@
     },
     {
         "name": "1080p video (muted)",
-        "url": "https://www.loom.com/share/313bf71d20ca47b2a35b6634cefdb761",
+        "url": "https://www.loom.com/share/d165fd054a294d8a8587807bcc50c885",
         "params": {
             "downloadMode": "mute"
         },
@@ -21,7 +21,7 @@
     },
     {
         "name": "1080p video (audio only)",
-        "url": "https://www.loom.com/share/313bf71d20ca47b2a35b6634cefdb761",
+        "url": "https://www.loom.com/share/d165fd054a294d8a8587807bcc50c885",
         "params": {
             "downloadMode": "audio"
         },
@@ -29,5 +29,32 @@
             "code": 400,
             "status": "error"
         }
+    },
+    {
+        "name": "video with no transcodedUrl",
+        "url": "https://www.loom.com/share/aa3d8b08bee74d05af5b42989e9f33e9",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "redirect"
+        }
+    },
+    {
+        "name": "video with title in url",
+        "url": "https://www.loom.com/share/Meet-AI-workflows-aa3d8b08bee74d05af5b42989e9f33e9",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "redirect"
+        }
+    },
+    {
+        "name": "video with title in url (2)",
+        "url": "https://www.loom.com/share/Unlocking-Incredible-Organizational-Velocity-with-Async-Video-4a2a8baf124c4390954dcbb46a58cfd7",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "redirect"
+        }
     }
 ]

api/src/util/tests/pinterest.json

@@ -54,7 +54,7 @@
         "params": {},
         "expected": {
             "code": 200,
-            "status": "redirect"
+            "status": "tunnel"
         }
     },
     {
@@ -63,25 +63,25 @@
         "params": {},
         "expected": {
             "code": 200,
-            "status": "redirect"
+            "status": "tunnel"
         }
     },
     {
         "name": "regular gif",
-        "url": "https://www.pinterest.com/pin/814447913881127862/",
+        "url": "https://www.pinterest.com/pin/643170390530326178/",
         "params": {},
         "expected": {
             "code": 200,
-            "status": "redirect"
+            "status": "tunnel"
         }
     },
     {
         "name": "regular gif (.ca TLD)",
-        "url": "https://www.pinterest.ca/pin/814447913881127862/",
+        "url": "https://www.pinterest.ca/pin/643170390530326178/",
         "params": {},
         "expected": {
             "code": 200,
-            "status": "redirect"
+            "status": "tunnel"
         }
     }
 ]

api/src/util/tests/reddit.json

@@ -56,5 +56,23 @@
             "code": 200,
             "status": "tunnel"
         }
+    },
+    {
+        "name": "shortened video link",
+        "url": "https://v.redd.it/ifg2emt5ck0e1",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
+    },
+    {
+        "name": "shortened video link (alternative)",
+        "url": "https://reddit.com/video/ifg2emt5ck0e1",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
     }
 ]

api/src/util/tests/soundcloud.json

@@ -59,7 +59,7 @@
     },
     {
         "name": "on.soundcloud link",
-        "url": "https://on.soundcloud.com/wLZre",
+        "url": "https://on.soundcloud.com/XHLLKSXRQ5yyGDuD9",
         "params": {},
         "expected": {
             "code": 200,

api/src/util/tests/twitter.json

@@ -102,9 +102,8 @@
     },
     {
         "name": "retweeted video",
-        "url": "https://twitter.com/uwukko/status/1696901469633421344",
+        "url": "https://twitter.com/schlizzawg/status/1869017025055793405",
         "params": {},
-        "canFail": true,
         "expected": {
             "code": 200,
             "status": "redirect"
@@ -145,7 +144,7 @@
         "params": {},
         "expected": {
             "code": 200,
-            "status": "redirect"
+            "status": "tunnel"
         }
     },
     {
@@ -170,6 +169,15 @@
             "status": "tunnel"
         }
     },
+    {
+        "name": "gif",
+        "url": "https://x.com/thelastromances/status/1897839691212202479",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
+    },
     {
         "name": "inexistent post",
         "url": "https://twitter.com/test/status/9487653",
@@ -203,7 +211,16 @@
     },
     {
         "name": "bookmarked photo",
-        "url": "https://twitter.com/i/bookmarks?post_id=1837430141179289876",
+        "url": "https://twitter.com/i/bookmarks?post_id=1887450602164396149",
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
+    },
+    {
+        "name": "video in an ad card",
+        "url": "https://x.com/igorbrigadir/status/1611399816487084033?s=46",
         "params": {},
         "expected": {
             "code": 200,

api/src/util/tests/xiaohongshu.json

@@ -0,0 +1,60 @@
+[
+    {
+        "name": "video (might have expired)",
+        "url": "https://www.xiaohongshu.com/explore/67cc17a3000000000e00726a?xsec_token=CBSFRtbF57so920elY1kbIX4fE1nhrwlpGZs9m6pIFpwo=",
+        "canFail": true,
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
+    },
+    {
+        "name": "picker with multiple live photos (might have expired)",
+        "url": "https://www.xiaohongshu.com/explore/67c691b4000000000d0159cc?xsec_token=CB8p1eyB5DiFkwlUpy1BTeVsI9oOve6ppNjuDzo8V8p5w=",
+        "canFail": true,
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "picker"
+        }
+    },
+    {
+        "name": "one photo (might have expired)",
+        "url": "https://www.xiaohongshu.com/explore/676e132d000000000b016f68?xsec_token=ABRv6LKzizOFeSaf2HnnBkdBqniB5Ak1fI8tMAHzO31jA",
+        "canFail": true,
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
+    },
+    {
+        "name": "short link (might have expired)",
+        "url": "https://xhslink.com/a/czn4z6c1tic4",
+        "canFail": true,
+        "params": {},
+        "expected": {
+            "code": 200,
+            "status": "tunnel"
+        }
+    },
+    {
+        "name": "wrong note id",
+        "url": "https://www.xiaohongshu.com/discovery/item/6789065911100000210035fc?source=webshare&xhsshare=pc_web&xsec_token=CBustnz_Twf1BSybpe5-D-BzUb-Bx28DPLb418TN9S9Kk&xsec_source=pc_share",
+        "params": {},
+        "expected": {
+            "code": 400,
+            "status": "error"
+        }
+    },
+    {
+        "name": "short link, wrong id",
+        "url": "https://xhslink.com/a/aaaaaa",
+        "params": {},
+        "expected": {
+            "code": 400,
+            "status": "error"
+        }
+    }
+]

api/src/util/tests/youtube.json

@@ -189,6 +189,7 @@
     {
         "name": "hls video (h264, 1440p)",
         "url": "https://www.youtube.com/watch?v=vPwaXytZcgI",
+        "canFail": true,
         "params": {
             "youtubeVideoCodec": "h264",
             "videoQuality": "1440",
@@ -202,6 +203,7 @@
     {
         "name": "hls video (vp9, 360p)",
         "url": "https://www.youtube.com/watch?v=vPwaXytZcgI",
+        "canFail": true,
         "params": {
             "youtubeVideoCodec": "vp9",
             "videoQuality": "360",
@@ -215,6 +217,7 @@
     {
         "name": "hls video (audio mode)",
         "url": "https://www.youtube.com/watch?v=vPwaXytZcgI",
+        "canFail": true,
         "params": {
             "downloadMode": "audio",
             "youtubeHLS": true
@@ -227,6 +230,7 @@
     {
         "name": "hls video (audio mode, best format)",
         "url": "https://www.youtube.com/watch?v=vPwaXytZcgI",
+        "canFail": true,
         "params": {
             "downloadMode": "audio",
             "youtubeHLS": true,

pnpm-lock.yaml

@@ -28,11 +28,8 @@ importers:
       dotenv:
         specifier: ^16.0.1
         version: 16.4.5
-      esbuild:
-        specifier: ^0.14.51
-        version: 0.14.54
       express:
-        specifier: ^4.21.1
+        specifier: ^4.21.2
         version: 4.21.2
       express-rate-limit:
         specifier: ^7.4.1
@@ -46,12 +43,12 @@ importers:
       ipaddr.js:
         specifier: 2.2.0
         version: 2.2.0
+      mime:
+        specifier: ^4.0.4
+        version: 4.0.4
       nanoid:
-        specifier: ^4.0.2
+        specifier: ^5.0.9
-        version: 4.0.2
+        version: 5.1.5
-      node-cache:
-        specifier: ^5.1.2
-        version: 5.1.2
       set-cookie-parser:
         specifier: 2.6.0
         version: 2.6.0
@@ -62,8 +59,8 @@ importers:
         specifier: 1.0.3
         version: 1.0.3
       youtubei.js:
-        specifier: ^12.2.0
+        specifier: ^13.4.0
-        version: 12.2.0
+        version: 13.4.0
       zod:
         specifier: ^3.23.8
         version: 3.23.8
@@ -336,12 +333,6 @@ packages:
     cpu: [ia32]
     os: [linux]
 
-  '@esbuild/linux-loong64@0.14.54':
-    resolution: {integrity: sha512-bZBrLAIX1kpWelV0XemxBZllyRmM6vgFQQG2GdNb+r3Fkp0FOh1NJSvekXDs7jq70k4euu1cryLMfU+mTXlEpw==}
-    engines: {node: '>=12'}
-    cpu: [loong64]
-    os: [linux]
-
   '@esbuild/linux-loong64@0.21.5':
     resolution: {integrity: sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==}
     engines: {node: '>=12'}
@@ -960,10 +951,6 @@ packages:
     resolution: {integrity: sha512-7VT13fmjotKpGipCW9JEQAusEPE+Ei8nl6/g4FBAmIm0GOOLMua9NDDo/DWp0ZAxCr3cPq5ZpBqmPAQgDda2Pw==}
     engines: {node: '>= 8.10.0'}
 
-  clone@2.1.2:
-    resolution: {integrity: sha512-3Pe/CF1Nn94hyhIYpjtiLhdCoEoz0DqQ+988E9gmeEdQZlojxnOb74wctFyuwWQHzqyf9X7C7MG8juUpqBJT8w==}
-    engines: {node: '>=0.8'}
-
   cluster-key-slot@1.1.2:
     resolution: {integrity: sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==}
     engines: {node: '>=0.10.0'}
@@ -1124,131 +1111,6 @@ packages:
   es6-promise@3.3.1:
     resolution: {integrity: sha512-SOp9Phqvqn7jtEUxPWdWfWoLmyt2VaJ6MpvP9Comy1MceMXqE6bxvaTu4iaxpYYPzhny28Lc+M87/c2cPK6lDg==}
 
-  esbuild-android-64@0.14.54:
-    resolution: {integrity: sha512-Tz2++Aqqz0rJ7kYBfz+iqyE3QMycD4vk7LBRyWaAVFgFtQ/O8EJOnVmTOiDWYZ/uYzB4kvP+bqejYdVKzE5lAQ==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [android]
-
-  esbuild-android-arm64@0.14.54:
-    resolution: {integrity: sha512-F9E+/QDi9sSkLaClO8SOV6etqPd+5DgJje1F9lOWoNncDdOBL2YF59IhsWATSt0TLZbYCf3pNlTHvVV5VfHdvg==}
-    engines: {node: '>=12'}
-    cpu: [arm64]
-    os: [android]
-
-  esbuild-darwin-64@0.14.54:
-    resolution: {integrity: sha512-jtdKWV3nBviOd5v4hOpkVmpxsBy90CGzebpbO9beiqUYVMBtSc0AL9zGftFuBon7PNDcdvNCEuQqw2x0wP9yug==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [darwin]
-
-  esbuild-darwin-arm64@0.14.54:
-    resolution: {integrity: sha512-OPafJHD2oUPyvJMrsCvDGkRrVCar5aVyHfWGQzY1dWnzErjrDuSETxwA2HSsyg2jORLY8yBfzc1MIpUkXlctmw==}
-    engines: {node: '>=12'}
-    cpu: [arm64]
-    os: [darwin]
-
-  esbuild-freebsd-64@0.14.54:
-    resolution: {integrity: sha512-OKwd4gmwHqOTp4mOGZKe/XUlbDJ4Q9TjX0hMPIDBUWWu/kwhBAudJdBoxnjNf9ocIB6GN6CPowYpR/hRCbSYAg==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [freebsd]
-
-  esbuild-freebsd-arm64@0.14.54:
-    resolution: {integrity: sha512-sFwueGr7OvIFiQT6WeG0jRLjkjdqWWSrfbVwZp8iMP+8UHEHRBvlaxL6IuKNDwAozNUmbb8nIMXa7oAOARGs1Q==}
-    engines: {node: '>=12'}
-    cpu: [arm64]
-    os: [freebsd]
-
-  esbuild-linux-32@0.14.54:
-    resolution: {integrity: sha512-1ZuY+JDI//WmklKlBgJnglpUL1owm2OX+8E1syCD6UAxcMM/XoWd76OHSjl/0MR0LisSAXDqgjT3uJqT67O3qw==}
-    engines: {node: '>=12'}
-    cpu: [ia32]
-    os: [linux]
-
-  esbuild-linux-64@0.14.54:
-    resolution: {integrity: sha512-EgjAgH5HwTbtNsTqQOXWApBaPVdDn7XcK+/PtJwZLT1UmpLoznPd8c5CxqsH2dQK3j05YsB3L17T8vE7cp4cCg==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [linux]
-
-  esbuild-linux-arm64@0.14.54:
-    resolution: {integrity: sha512-WL71L+0Rwv+Gv/HTmxTEmpv0UgmxYa5ftZILVi2QmZBgX3q7+tDeOQNqGtdXSdsL8TQi1vIaVFHUPDe0O0kdig==}
-    engines: {node: '>=12'}
-    cpu: [arm64]
-    os: [linux]
-
-  esbuild-linux-arm@0.14.54:
-    resolution: {integrity: sha512-qqz/SjemQhVMTnvcLGoLOdFpCYbz4v4fUo+TfsWG+1aOu70/80RV6bgNpR2JCrppV2moUQkww+6bWxXRL9YMGw==}
-    engines: {node: '>=12'}
-    cpu: [arm]
-    os: [linux]
-
-  esbuild-linux-mips64le@0.14.54:
-    resolution: {integrity: sha512-qTHGQB8D1etd0u1+sB6p0ikLKRVuCWhYQhAHRPkO+OF3I/iSlTKNNS0Lh2Oc0g0UFGguaFZZiPJdJey3AGpAlw==}
-    engines: {node: '>=12'}
-    cpu: [mips64el]
-    os: [linux]
-
-  esbuild-linux-ppc64le@0.14.54:
-    resolution: {integrity: sha512-j3OMlzHiqwZBDPRCDFKcx595XVfOfOnv68Ax3U4UKZ3MTYQB5Yz3X1mn5GnodEVYzhtZgxEBidLWeIs8FDSfrQ==}
-    engines: {node: '>=12'}
-    cpu: [ppc64]
-    os: [linux]
-
-  esbuild-linux-riscv64@0.14.54:
-    resolution: {integrity: sha512-y7Vt7Wl9dkOGZjxQZnDAqqn+XOqFD7IMWiewY5SPlNlzMX39ocPQlOaoxvT4FllA5viyV26/QzHtvTjVNOxHZg==}
-    engines: {node: '>=12'}
-    cpu: [riscv64]
-    os: [linux]
-
-  esbuild-linux-s390x@0.14.54:
-    resolution: {integrity: sha512-zaHpW9dziAsi7lRcyV4r8dhfG1qBidQWUXweUjnw+lliChJqQr+6XD71K41oEIC3Mx1KStovEmlzm+MkGZHnHA==}
-    engines: {node: '>=12'}
-    cpu: [s390x]
-    os: [linux]
-
-  esbuild-netbsd-64@0.14.54:
-    resolution: {integrity: sha512-PR01lmIMnfJTgeU9VJTDY9ZerDWVFIUzAtJuDHwwceppW7cQWjBBqP48NdeRtoP04/AtO9a7w3viI+PIDr6d+w==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [netbsd]
-
-  esbuild-openbsd-64@0.14.54:
-    resolution: {integrity: sha512-Qyk7ikT2o7Wu76UsvvDS5q0amJvmRzDyVlL0qf5VLsLchjCa1+IAvd8kTBgUxD7VBUUVgItLkk609ZHUc1oCaw==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [openbsd]
-
-  esbuild-sunos-64@0.14.54:
-    resolution: {integrity: sha512-28GZ24KmMSeKi5ueWzMcco6EBHStL3B6ubM7M51RmPwXQGLe0teBGJocmWhgwccA1GeFXqxzILIxXpHbl9Q/Kw==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [sunos]
-
-  esbuild-windows-32@0.14.54:
-    resolution: {integrity: sha512-T+rdZW19ql9MjS7pixmZYVObd9G7kcaZo+sETqNH4RCkuuYSuv9AGHUVnPoP9hhuE1WM1ZimHz1CIBHBboLU7w==}
-    engines: {node: '>=12'}
-    cpu: [ia32]
-    os: [win32]
-
-  esbuild-windows-64@0.14.54:
-    resolution: {integrity: sha512-AoHTRBUuYwXtZhjXZbA1pGfTo8cJo3vZIcWGLiUcTNgHpJJMC1rVA44ZereBHMJtotyN71S8Qw0npiCIkW96cQ==}
-    engines: {node: '>=12'}
-    cpu: [x64]
-    os: [win32]
-
-  esbuild-windows-arm64@0.14.54:
-    resolution: {integrity: sha512-M0kuUvXhot1zOISQGXwWn6YtS+Y/1RT9WrVIOywZnJHo3jCDyewAc79aKNQWFCQm+xNHVTq9h8dZKvygoXQQRg==}
-    engines: {node: '>=12'}
-    cpu: [arm64]
-    os: [win32]
-
-  esbuild@0.14.54:
-    resolution: {integrity: sha512-Cy9llcy8DvET5uznocPyqL3BFRrFXSVqbgpMJ9Wz8oVjZlh/zUSNbPRbov0VX7VxN2JH1Oa0uNxZ7eLRb62pJA==}
-    engines: {node: '>=12'}
-    hasBin: true
-
   esbuild@0.21.5:
     resolution: {integrity: sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==}
     engines: {node: '>=12'}
@@ -1555,8 +1417,8 @@ packages:
   jackspeak@3.4.3:
     resolution: {integrity: sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw==}
 
-  jintr@3.2.0:
+  jintr@3.3.1:
-    resolution: {integrity: sha512-psD1yf05kMKDNsUdW1l5YhO59pHScQ6OIHHb8W5SKSM2dCOFPsqolmIuSHgVA8+3Dc47NJR181CXZ4alCAPTkA==}
+    resolution: {integrity: sha512-nnOzyhf0SLpbWuZ270Omwbj5LcXUkTcZkVnK8/veJXtSZOiATM5gMZMdmzN75FmTyj+NVgrGaPdH12zIJ24oIA==}
 
   joycon@3.1.1:
     resolution: {integrity: sha512-34wB/Y7MW7bzjKRjUKTa46I2Z7eV62Rkhva+KkopW7Qvv/OSWBqvkSY7vusOPrNuZcUG3tApvdVgNB8POj3SPw==}
@@ -1715,9 +1577,9 @@ packages:
     engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1}
     hasBin: true
 
-  nanoid@4.0.2:
+  nanoid@5.1.5:
-    resolution: {integrity: sha512-7ZtY5KTCNheRGfEFxnedV5zFiORN1+Y1N6zvPTnHQd8ENUvfaDBeuJDZb2bN/oXwXxu3qkTXDzy57W5vAmDTBw==}
+    resolution: {integrity: sha512-Ir/+ZpE9fDsNH0hQ3C68uyThDXzYcim2EqcZ8zn8Chtt1iylPT9xXJB0kPCnqzgcEGikO9RxSrh63MsmVCU7Fw==}
-    engines: {node: ^14 || ^16 || >=18}
+    engines: {node: ^18 || >=20}
     hasBin: true
 
   natural-compare@1.4.0:
@@ -1727,10 +1589,6 @@ packages:
     resolution: {integrity: sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==}
     engines: {node: '>= 0.6'}
 
-  node-cache@5.1.2:
-    resolution: {integrity: sha512-t1QzWwnk4sjLWaQAS8CHgOJ+RAfmHpxFWmc36IWTiWHQfs0w5JDMBS1b1ZxQteo0vVVuWJvIUKHDkkeK7vIGCg==}
-    engines: {node: '>= 8.0.0'}
-
   normalize-path@3.0.0:
     resolution: {integrity: sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==}
     engines: {node: '>=0.10.0'}
@@ -2354,8 +2212,8 @@ packages:
     resolution: {integrity: sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==}
     engines: {node: '>=10'}
 
-  youtubei.js@12.2.0:
+  youtubei.js@13.4.0:
-    resolution: {integrity: sha512-G+50qrbJCToMYhu8jbaHiS3Vf+RRul+CcDbz3hEGwHkGPh+zLiWwD6SS+YhYF+2/op4ZU5zDYQJrGqJ+wKh7Gw==}
+    resolution: {integrity: sha512-+fmIZU/dWAjsROONrASy1REwVpy6umAPVuoNLr/4iNmZXl84LyBef0n3hrd1Vn9035EuINToGyQcBmifwUEemA==}
 
   zod@3.23.8:
     resolution: {integrity: sha512-XBx9AXhXktjUqnepgTiE5flcKIYWi/rme0Eaj+5Y0lftuGBq+jyRu/md4WnuxqgP1ubdpNCsYEYPxrzVHD8d6g==}
@@ -2448,9 +2306,6 @@ snapshots:
   '@esbuild/linux-ia32@0.23.0':
     optional: true
 
-  '@esbuild/linux-loong64@0.14.54':
-    optional: true
-
   '@esbuild/linux-loong64@0.21.5':
     optional: true
 
@@ -3003,8 +2858,6 @@ snapshots:
     optionalDependencies:
       fsevents: 2.3.3
 
-  clone@2.1.2: {}
-
   cluster-key-slot@1.1.2:
     optional: true
 
@@ -3127,90 +2980,6 @@ snapshots:
 
   es6-promise@3.3.1: {}
 
-  esbuild-android-64@0.14.54:
-    optional: true
-
-  esbuild-android-arm64@0.14.54:
-    optional: true
-
-  esbuild-darwin-64@0.14.54:
-    optional: true
-
-  esbuild-darwin-arm64@0.14.54:
-    optional: true
-
-  esbuild-freebsd-64@0.14.54:
-    optional: true
-
-  esbuild-freebsd-arm64@0.14.54:
-    optional: true
-
-  esbuild-linux-32@0.14.54:
-    optional: true
-
-  esbuild-linux-64@0.14.54:
-    optional: true
-
-  esbuild-linux-arm64@0.14.54:
-    optional: true
-
-  esbuild-linux-arm@0.14.54:
-    optional: true
-
-  esbuild-linux-mips64le@0.14.54:
-    optional: true
-
-  esbuild-linux-ppc64le@0.14.54:
-    optional: true
-
-  esbuild-linux-riscv64@0.14.54:
-    optional: true
-
-  esbuild-linux-s390x@0.14.54:
-    optional: true
-
-  esbuild-netbsd-64@0.14.54:
-    optional: true
-
-  esbuild-openbsd-64@0.14.54:
-    optional: true
-
-  esbuild-sunos-64@0.14.54:
-    optional: true
-
-  esbuild-windows-32@0.14.54:
-    optional: true
-
-  esbuild-windows-64@0.14.54:
-    optional: true
-
-  esbuild-windows-arm64@0.14.54:
-    optional: true
-
-  esbuild@0.14.54:
-    optionalDependencies:
-      '@esbuild/linux-loong64': 0.14.54
-      esbuild-android-64: 0.14.54
-      esbuild-android-arm64: 0.14.54
-      esbuild-darwin-64: 0.14.54
-      esbuild-darwin-arm64: 0.14.54
-      esbuild-freebsd-64: 0.14.54
-      esbuild-freebsd-arm64: 0.14.54
-      esbuild-linux-32: 0.14.54
-      esbuild-linux-64: 0.14.54
-      esbuild-linux-arm: 0.14.54
-      esbuild-linux-arm64: 0.14.54
-      esbuild-linux-mips64le: 0.14.54
-      esbuild-linux-ppc64le: 0.14.54
-      esbuild-linux-riscv64: 0.14.54
-      esbuild-linux-s390x: 0.14.54
-      esbuild-netbsd-64: 0.14.54
-      esbuild-openbsd-64: 0.14.54
-      esbuild-sunos-64: 0.14.54
-      esbuild-windows-32: 0.14.54
-      esbuild-windows-64: 0.14.54
-      esbuild-windows-arm64: 0.14.54
-
   esbuild@0.21.5:
     optionalDependencies:
       '@esbuild/aix-ppc64': 0.21.5
@@ -3639,7 +3408,7 @@ snapshots:
     optionalDependencies:
       '@pkgjs/parseargs': 0.11.0
 
-  jintr@3.2.0:
+  jintr@3.3.1:
     dependencies:
       acorn: 8.12.1
 
@@ -3761,16 +3530,12 @@ snapshots:
 
   nanoid@3.3.7: {}
 
-  nanoid@4.0.2: {}
+  nanoid@5.1.5: {}
 
   natural-compare@1.4.0: {}
 
   negotiator@0.6.3: {}
 
-  node-cache@5.1.2:
-    dependencies:
-      clone: 2.1.2
-
   normalize-path@3.0.0: {}
 
   npm-run-path@4.0.1:
@@ -4337,10 +4102,10 @@ snapshots:
 
   yocto-queue@0.1.0: {}
 
-  youtubei.js@12.2.0:
+  youtubei.js@13.4.0:
     dependencies:
       '@bufbuild/protobuf': 2.2.3
-      jintr: 3.2.0
+      jintr: 3.3.1
       tslib: 2.6.3
       undici: 5.28.4