Merge branch 'wukko:current' into axui2-patch-1

README.md

@@ -1,7 +1,7 @@
 # cobalt
 best way to save what you love: [cobalt.tools](https://cobalt.tools/)  
 
-![cobalt logo with repeated logo (double arrow) pattern background](https://raw.githubusercontent.com/wukko/cobalt/current/src/front/icons/pattern.png "cobalt logo with repeated logo (double arrow) pattern background")  
+![cobalt logo with repeated logo (double arrow) pattern background](/src/front/icons/pattern.png "cobalt logo with repeated logo (double arrow) pattern background")  
 
 ## what's cobalt?
 cobalt is a media downloader that doesn't piss you off. it's fast, friendly, and doesn't have any bullshit that modern web is filled with: ***no ads, trackers, or invasive analytics***.  
@@ -51,12 +51,15 @@ this list is not final and keeps expanding over time. if support for a service y
 | youtube    | supports videos, music, and shorts. 8K, 4K, HDR, VR, and high FPS videos. rich metadata & dubs. h264/av1/vp9 codecs. |
 
 ## cobalt api
-cobalt has an open api that you can use in projects *for completely free~*. it's easy and straightforward to use, [check out the docs](https://github.com/wukko/cobalt/blob/current/docs/api.md) to learn how to use it. 
+cobalt has an open api that you can use in your projects *for free~*. it's easy and straightforward to use, [check out the docs](/docs/api.md) to learn how to use it. 
 
-you can use the main api instance ([co.wuk.sh](https://co.wuk.sh/)) in your projects.
+✅ you can use the main api instance ([co.wuk.sh](https://co.wuk.sh/)) in your **personal** projects.  
+❌ you cannot use the free api commercially (anywhere that's gated behind paywalls or ads). host your own instance for this.
+
+we reserve the right to restrict abusive/excessive access to the main instance api.
 
 ## how to run your own instance
-if you want to run your own instance for whatever purpose, [follow this guide](https://github.com/wukko/cobalt/blob/current/docs/run-an-instance.md).  
+if you want to run your own instance for whatever purpose, [follow this guide](/docs/run-an-instance.md).  
 it's *highly* recommended to use a docker compose method unless you run for developing/debugging purposes.
 
 ## sponsors 
@@ -68,7 +71,7 @@ cobalt is a tool for easing content downloads from internet and takes ***zero li
 cobalt is ***NOT*** a piracy tool and cannot be used as such. it can only download free, publicly accessible content. such content can be easily downloaded through any browser's dev tools. pressing one button is easier, so i made a convenient, ad-less tool for such repeated actions.
 
 ## cobalt license
-cobalt code is licensed under [AGPL-3.0](https://github.com/wukko/cobalt/blob/current/LICENSE).
+cobalt code is licensed under [AGPL-3.0](/LICENSE).
 
 cobalt branding, mascots, and other related assets included in the repo are ***copyrighted*** and not covered by the AGPL-3.0 license. you ***cannot*** use them under same terms.  
 

docs/api.md

@@ -59,7 +59,7 @@ from a successful call to `/api/json`. however, the parameters passed to it are
 and **unmodifiable** from your (the api client's) perspective, and can change between versions.
 
 therefore you don't need to worry about what they mean - but if you really want to know, you can
-[read the source code](../src/modules/stream/manage.js).
+[read the source code](/src/modules/stream/manage.js).
 
 ## GET: `/api/serverInfo`
 returns current basic server info.  

docs/examples/docker-compose.example.yml

@@ -21,6 +21,8 @@ services:
             API_URL: "https://co.wuk.sh/"
             # replace eu-nl with your instance's distinctive name
             API_NAME: "eu-nl"
+            # if you want to use cookies when fetching data from services, uncomment the next line and the lines under volume
+            # COOKIE_PATH: "/cookies.json"
             # see docs/run-an-instance.md for more information
         labels:
             - com.centurylinklabs.watchtower.scope=cobalt

docs/run-an-instance.md

@@ -18,7 +18,7 @@ if you need help with installing docker, follow *only the first step* of these t
     ```  
     i'm using `nano` in this example, it may not be available in your distro. you can use any other text editor.  
 
-3. copy and paste the [sample config from here](https://github.com/wukko/cobalt/blob/current/docs/examples/docker-compose.example.yml) for either web or api instance (or both, if you wish) and edit it to your needs.  
+3. copy and paste the [sample config from here](examples/docker-compose.example.yml) for either web or api instance (or both, if you wish) and edit it to your needs.  
     make sure to replace default URLs with your own or cobalt won't work correctly.  
 
 4. finally, start the cobalt container (from cobalt directory):
@@ -26,7 +26,7 @@ if you need help with installing docker, follow *only the first step* of these t
     docker compose up -d
     ```
 
-if you want your instance to support services that require authentication to view public content, create `cookies.json` file in the same directory as `docker-compose.yml`. example cookies file [can be found here](https://github.com/wukko/cobalt/blob/current/docs/examples/cookies.example.json).
+if you want your instance to support services that require authentication to view public content, create `cookies.json` file in the same directory as `docker-compose.yml`. example cookies file [can be found here](examples/cookies.example.json).
 
 cobalt package will update automatically thanks to watchtower.
 

docs/troubleshooting.md

@@ -5,29 +5,33 @@
 if any issues occur while using cobalt, you can fix many of them yourself. this document aims to provide guides on how to fix most complicated of them.  
 use wiki navigation on right to jump between solutions.  
 
-## how to fix clipboard pasting in firefox
+## how to fix clipboard pasting in older versions of firefox
+```
+🎉 firefox finally supports pasting by default starting from version 125.
+
+👍 you don't need to follow this tutorial if you're on the latest version of firefox.
+```
 you can fix this issue by changing a single preference in `about:config`.
 
 ### steps to enable clipboard functionality
 1. go to `about:config`:  
-
-   ![screenshot showing about:config entered into address bar](https://github.com/wukko/cobalt/assets/71202418/9ad78612-a372-4949-aeac-99dfc41e273c)  
+   ![screenshot showing about:config entered into address bar](images/troubleshooting/clipboard/config.png)  
 
 2. if asked, read what firefox has to say and press "accept the risk and continue".  
  ⚠ tinkering with other preferences may break your browser. **do not** edit them unless you know what you're doing.   
 
-   ![screenshot showing about:config security warning that reads: "proceed with caution. changing advanced configuration preferences can impact firefox performance or security." lower there's a pre-checked checkbox that says: "warn me when i attempt to access these preferences". lowest element is a blue button that says "accept the risk and continue"](https://github.com/wukko/cobalt/assets/71202418/02328729-dbfe-4ea4-b2ca-7bcf1998c2ca)  
+   ![screenshot showing about:config security warning that reads: "proceed with caution. changing advanced configuration preferences can impact firefox performance or security." lower there's a pre-checked checkbox that says: "warn me when i attempt to access these preferences". lowest element is a blue button that says "accept the risk and continue"](images/troubleshooting/clipboard/risk.png)  
 
 3. search for `dom.events.asyncClipboard.readText`  
 
-   ![screenshot showing "dom.events.asyncclipboard.readtext" entered into search on about:config page](https://github.com/wukko/cobalt/assets/71202418/7c7f7e3c-6a6a-40df-8436-277489e72e0b)
+   ![screenshot showing "dom.events.asyncclipboard.readtext" entered into search on about:config page](images/troubleshooting/clipboard/search.png)
 
 4. press the toggle button on very right.  
 
-   ![screenshot showing "dom.events.asyncclipboard.readtext" preference on about:config page with highlighted toggle button on very right](https://github.com/wukko/cobalt/assets/71202418/b45db18e-f4bf-4f1c-9a8c-f13a63a21335)
+   ![screenshot showing "dom.events.asyncclipboard.readtext" preference on about:config page with highlighted toggle button on very right](images/troubleshooting/clipboard/toggle.png)
 
 5. "false" should change to "true".  
 
-   ![screenshot showing "dom.events.asyncclipboard.readtext" preference on about:config page, this one with "true" text highlighted](https://github.com/wukko/cobalt/assets/71202418/4869b4ff-8385-4cd3-ae59-aa2e03a58b5f)
+   ![screenshot showing "dom.events.asyncclipboard.readtext" preference on about:config page, this one with "true" text highlighted](images/troubleshooting/clipboard/toggled.png)
 
-6. go back to cobalt, reload the page, press `paste and download` button again. this time it works! enjoy simpler downloading experience :)
+6. go back to cobalt, reload the page, press `paste` button again. this time it works! enjoy simpler downloading experience :)

package.json

@@ -1,7 +1,7 @@
 {
     "name": "cobalt",
     "description": "save what you love",
-    "version": "7.12.4",
+    "version": "7.12.6",
     "author": "wukko",
     "exports": "./src/cobalt.js",
     "type": "module",

src/front/manifest.webmanifest

@@ -1 +1,75 @@
-{"name":"cobalt","short_name":"cobalt","start_url":"/","icons":[{"src":"/icons/android-chrome-192x192.png","sizes":"192x192","type":"image/png"},{"src":"/icons/android-chrome-512x512.png","sizes":"512x512","type":"image/png"},{"src":"/icons/generic.png","sizes":"512x512","type":"image/png","purpose":"any"},{"src":"/icons/maskable/x48.png","sizes":"48x48","type":"image/png","purpose":"maskable"},{"src":"/icons/maskable/x72.png","sizes":"72x72","type":"image/png","purpose":"maskable"},{"src":"/icons/maskable/x96.png","sizes":"96x96","type":"image/png","purpose":"maskable"},{"src":"/icons/maskable/x128.png","sizes":"128x128","type":"image/png","purpose":"maskable"},{"src":"/icons/maskable/x192.png","sizes":"192x192","type":"image/png","purpose":"maskable"},{"src":"/icons/maskable/x384.png","sizes":"384x384","type":"image/png","purpose":"maskable"},{"src":"/icons/maskable/x512.png","sizes":"512x512","type":"image/png","purpose":"maskable"},{"src":"/icons/maskable/x1280.png","sizes":"1280x1280","type":"image/png","purpose":"maskable"}],"theme_color":"#000000","background_color":"#000000","display":"standalone"}
+{
+    "name": "cobalt",
+    "short_name": "cobalt",
+    "start_url": "/",
+    "icons": [
+        {
+            "src": "/icons/android-chrome-192x192.png",
+            "sizes": "192x192",
+            "type": "image/png"
+        },
+        {
+            "src": "/icons/android-chrome-512x512.png",
+            "sizes": "512x512",
+            "type": "image/png"
+        },
+        {
+            "src": "/icons/generic.png",
+            "sizes": "512x512",
+            "type": "image/png",
+            "purpose": "any"
+        },
+        {
+            "src": "/icons/maskable/48.png",
+            "sizes": "48x48",
+            "type": "image/png",
+            "purpose": "maskable"
+        },
+        {
+            "src": "/icons/maskable/72.png",
+            "sizes": "72x72",
+            "type": "image/png",
+            "purpose": "maskable"
+        },
+        {
+            "src": "/icons/maskable/96.png",
+            "sizes": "96x96",
+            "type": "image/png",
+            "purpose": "maskable"
+        },
+        {
+            "src": "/icons/maskable/128.png",
+            "sizes": "128x128",
+            "type": "image/png",
+            "purpose": "maskable"
+        },
+        {
+            "src": "/icons/maskable/192.png",
+            "sizes": "192x192",
+            "type": "image/png",
+            "purpose": "maskable"
+        },
+        {
+            "src": "/icons/maskable/384.png",
+            "sizes": "384x384",
+            "type": "image/png",
+            "purpose": "maskable"
+        },
+        {
+            "src": "/icons/maskable/512.png",
+            "sizes": "512x512",
+            "type": "image/png",
+            "purpose": "maskable"
+        }
+    ],
+    "share_target": {
+        "action": "/",
+        "params": {
+            "text": "u",
+            "url": "u"
+        }
+    },
+    "theme_color": "#000000",
+    "background_color": "#000000",
+    "display": "standalone"
+}

src/modules/processing/services/instagram.js

@@ -3,28 +3,55 @@ import { genericUserAgent } from "../../config.js";
 import { getCookie, updateCookie } from "../cookie/manager.js";
 
 const commonInstagramHeaders = {
-    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
-    'User-Agent': genericUserAgent,
-    'X-Ig-App-Id': '936619743392459',
-    'X-Asbd-Id': '129477',
-    'x-requested-with': 'XMLHttpRequest',
-    'Sec-Fetch-Dest': 'empty',
-    'Sec-Fetch-Mode': 'cors',
-    'Sec-Fetch-Site': 'same-origin',
-    'upgrade-insecure-requests': '1',
-    'accept-encoding': 'gzip, deflate, br',
-    'accept-language': 'en-US,en;q=0.9,en;q=0.8',
+    'user-agent': genericUserAgent,
+    'sec-gpc': '1',
+    'sec-fetch-site': 'same-origin',
+    'x-ig-app-id': '936619743392459'
 }
 
-async function request(url, cookie) {
+const cachedDtsg = {
+    value: '',
+    expiry: 0
+}
+
+async function findDtsgId(cookie) {
+    try {
+        if (cachedDtsg.expiry > Date.now()) return cachedDtsg.value;
+
+        const data = await fetch('https://www.instagram.com/', {
+            headers: {
+                ...commonInstagramHeaders,
+                cookie
+            }
+        }).then(r => r.text());
+
+        const token = data.match(/"dtsg":{"token":"(.*?)"/)[1];
+
+        cachedDtsg.value = token;
+        cachedDtsg.expiry = Date.now() + 86390000;
+
+        if (token) return token;
+        return false;
+    }
+    catch {}
+}
+
+async function request(url, cookie, method = 'GET', requestData) {
+    let headers = {
+        ...commonInstagramHeaders,
+        'x-ig-www-claim': cookie?._wwwClaim || '0',
+        'x-csrftoken': cookie?.values()?.csrftoken,
+        cookie
+    }
+    if (method === 'POST') {
+        headers['content-type'] = 'application/x-www-form-urlencoded';
+    }
+
     const data = await fetch(url, {
-        headers: {
-            ...commonInstagramHeaders,
-            'x-ig-www-claim': cookie?._wwwClaim || '0',
-            'x-csrftoken': cookie?.values()?.csrftoken,
-            cookie
-        }
-    })
+        method,
+        headers,
+        body: requestData && new URLSearchParams(requestData),
+    });
 
     if (data.headers.get('X-Ig-Set-Www-Claim') && cookie)
         cookie._wwwClaim = data.headers.get('X-Ig-Set-Www-Claim');
@@ -33,28 +60,64 @@ async function request(url, cookie) {
     return data.json();
 }
 
-async function getPost(id) {
-    let data;
-    try {
-        const cookie = getCookie('instagram');
+async function requestHTML(id, cookie) {
+    const data = await fetch(`https://www.instagram.com/p/${id}/embed/captioned/`, {
+        headers: {
+            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
+            "Accept-Language": "en-GB,en;q=0.9",
+            "Cache-Control": "max-age=0",
+            "Dnt": "1",
+            "Priority": "u=0, i",
+            "Sec-Ch-Ua": 'Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99',
+            "Sec-Ch-Ua-Mobile": "?0",
+            "Sec-Ch-Ua-Platform": "macOS",
+            "Sec-Fetch-Dest": "document",
+            "Sec-Fetch-Mode": "navigate",
+            "Sec-Fetch-Site": "none",
+            "Sec-Fetch-User": "?1",
+            "Upgrade-Insecure-Requests": "1",
+            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
+            ...cookie
+        }
+    }).then(r => r.text());
 
-        const url = new URL('https://www.instagram.com/graphql/query/');
-        url.searchParams.set('query_hash', 'b3055c01b4b222b8a47dc12b090e4e64')
-        url.searchParams.set('variables', JSON.stringify({
-            child_comment_count: 3,
-            fetch_comment_count: 40,
-            has_threaded_comments: true,
-            parent_comment_count: 24,
-            shortcode: id
-        }))
+    let embedData = JSON.parse(data?.match(/"init",\[\],\[(.*?)\]\],/)[1]);
 
-        data = (await request(url, cookie)).data;
+    if (!embedData || !embedData?.contextJSON) return false;
 
-    } catch {}
+    embedData = JSON.parse(embedData.contextJSON);
 
-    if (!data) return { error: 'ErrorCouldntFetch' };
+    return embedData;
+}
+async function requestGQL(id, cookie) {
+    let dtsgId;
 
-    const sidecar = data?.shortcode_media?.edge_sidecar_to_children;
+    if (cookie) {
+        dtsgId = await findDtsgId(cookie);
+    }
+    const url = new URL('https://www.instagram.com/api/graphql/');
+
+    const requestData = {
+        jazoest: '26406',
+        variables: JSON.stringify({
+          shortcode: id,
+          __relay_internal__pv__PolarisShareMenurelayprovider: false
+        }),
+        doc_id: '7153618348081770'
+    };
+    if (dtsgId) {
+        requestData.fb_dtsg = dtsgId;
+    }
+
+    return (await request(url, cookie, 'POST', requestData))
+            .data
+            ?.xdt_api__v1__media__shortcode__web_info
+            ?.items
+            ?.[0];
+}
+
+function extractOldPost(data, id) {
+    const sidecar = data?.gql_data?.shortcode_media?.edge_sidecar_to_children;
     if (sidecar) {
         const picker = sidecar.edges.filter(e => e.node?.display_url)
             .map(e => {
@@ -75,19 +138,86 @@ async function getPost(id) {
             });
 
         if (picker.length) return { picker }
-    } else if (data?.shortcode_media?.video_url) {
+    } else if (data?.gql_data?.shortcode_media?.video_url) {
         return {
-            urls: data.shortcode_media.video_url,
+            urls: data.gql_data.shortcode_media.video_url,
             filename: `instagram_${id}.mp4`,
             audioFilename: `instagram_${id}_audio`
         }
-    } else if (data?.shortcode_media?.display_url) {
+    } else if (data?.gql_data?.shortcode_media?.display_url) {
         return {
-            urls: data.shortcode_media.display_url,
+            urls: data.gql_data?.shortcode_media.display_url,
             isPhoto: true
         }
     }
+}
 
+function extractNewPost(data, id) {
+    const carousel = data.carousel_media;
+    if (carousel) {
+        const picker = carousel.filter(e => e?.image_versions2)
+            .map(e => {
+                const type = e.video_versions ? "video" : "photo";
+                const imageUrl = e.image_versions2.candidates[0].url;
+                
+                let url = imageUrl;
+                if (type === 'video') {
+                    const video = e.video_versions.reduce((a, b) => a.width * a.height < b.width * b.height ? b : a);
+                    url = video.url;
+                }
+
+                return {
+                    type, url,
+                    /* thumbnails have `Cross-Origin-Resource-Policy`
+                    ** set to `same-origin`, so we need to proxy them */
+                    thumb: createStream({
+                        service: "instagram",
+                        type: "default",
+                        u: imageUrl,
+                        filename: "image.jpg"
+                    })
+                }
+            });
+
+        if (picker.length) return { picker }
+    } else if (data.video_versions) {
+        const video = data.video_versions.reduce((a, b) => a.width * a.height < b.width * b.height ? b : a)
+        return {
+            urls: video.url,
+            filename: `instagram_${id}.mp4`,
+            audioFilename: `instagram_${id}_audio`
+        }
+    } else if (data.image_versions2?.candidates) {
+        return {
+            urls: data.image_versions2.candidates[0].url,
+            isPhoto: true
+        }
+    }
+}
+
+async function getPost(id) {
+    let data, result, dataType = 'old';
+    try {
+        const cookie = getCookie('instagram');
+
+        data = await requestHTML(id);
+        if (!data && cookie) data = await requestHTML(id, cookie);
+
+        if (!data) {
+            dataType = 'new';
+            data = await requestGQL(id, cookie);
+        }
+    } catch {}
+
+    if (!data) return { error: 'ErrorCouldntFetch' };
+
+    if (dataType === 'new') {
+        result = extractNewPost(data, id)
+    } else {
+        result = extractOldPost(data, id)
+    }
+
+    if (result) return result;
     return { error: 'ErrorEmptyDownload' }
 }
 
@@ -103,22 +233,31 @@ async function usernameToId(username, cookie) {
 
 async function getStory(username, id) {
     const cookie = getCookie('instagram');
-    if (!cookie) return { error: 'ErrorUnsupported' }
+    if (!cookie) return { error: 'ErrorUnsupported' };
 
     const userId = await usernameToId(username, cookie);
-    if (!userId) return { error: 'ErrorEmptyDownload' }
+    if (!userId) return { error: 'ErrorEmptyDownload' };
+    
+    const dtsgId = await findDtsgId(cookie);
 
-    const url = new URL('https://www.instagram.com/api/v1/feed/reels_media/');
-          url.searchParams.set('reel_ids', userId);
-          url.searchParams.set('media_id', id);
+    const url = new URL('https://www.instagram.com/api/graphql/');
+    const requestData = {
+        fb_dtsg: dtsgId,
+        jazoest: '26438',
+        variables: JSON.stringify({
+            reel_ids_arr : [ userId ],
+        }),
+        server_timestamps: true,
+        doc_id: '25317500907894419'
+    };
 
     let media;
     try {
-        const data = await request(url, cookie);
-        media = data?.reels_media?.find(m => m.id === userId);
+        const data = (await request(url, cookie, 'POST', requestData));
+        media = data?.data?.xdt_api__v1__feed__reels_media?.reels_media?.find(m => m.id === userId);
     } catch {}
 
-    const item = media.items[media.media_ids.indexOf(id)];
+    const item = media.items.find(m => m.pk === id);
     if (!item) return { error: 'ErrorEmptyDownload' };
     
     if (item.video_versions) {

src/modules/processing/services/soundcloud.js

@@ -1,7 +1,10 @@
 import { maxVideoDuration } from "../../config.js";
 import { cleanString } from "../../sub/utils.js";
 
-let cachedID = {};
+const cachedID = {
+    version: '',
+    id: ''
+};
 
 async function findClientID() {
     try {

src/modules/processing/services/youtube.js

@@ -33,7 +33,7 @@ export default async function(o) {
     }
 
     try {
-        info = await yt.getBasicInfo(o.id, 'ANDROID');
+        info = await yt.getBasicInfo(o.id, 'YTMUSIC_ANDROID');
     } catch (e) {
         return { error: 'ErrorCantConnectToServiceAPI' };
     }

src/modules/processing/servicesConfig.json

@@ -79,7 +79,7 @@
         "instagram": {
             "alias": "instagram reels, posts & stories",
             "patterns": [
-                "reels/:postId", "reel/:postId", "p/:postId", ":username/p/:postId",
+                "reels/:postId", ":username/reel/:postId", "reel/:postId", "p/:postId", ":username/p/:postId",
                 "tv/:postId", "stories/:username/:storyId"
             ],
             "enabled": true