CypherpunkSamurai/NewPipeExtractor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
2110 commits 2 branches 0 tags 95 MiB
Commit graph

2110 commits

This branch
This branch
All branches
Author SHA1 Message Date
litetex
0beb55a232 Improve `YoutubeThrottlingDecrypterTest` 2022-08-21 20:17:05 +02:00
litetex
9e93d6b193 YoutubeThrottlingDecrypter: Check if returned string is a valid JavaScript function 2022-08-21 20:16:45 +02:00
litetex
ed0a07af4e YoutubeThrottlingDecrypter: Patch regex 2022-08-21 20:15:53 +02:00
litetex
641a447d9b Updated consent-related mock-data 2022-08-21 18:43:48 +02:00
litetex
8ff7a90f52 Improved consent cookie related constants and documentation 2022-08-21 18:41:40 +02:00
litetex
5bbea3a8f2 Fix `YoutubeMixPlaylistExtractorTest` again 
Doesn't work for kurgesagt videos anymore, used music video (copyright free) instead
 2022-08-14 15:12:27 +02:00
litetex
da06166065 Updated mock data of `YoutubeSearchExtractorTest$Suggestion` 2022-08-14 15:12:26 +02:00
litetex
2e36ab1578 Fixed `YoutubeSearchExtractorTest$Suggestion` 2022-08-14 15:12:26 +02:00
litetex
2a8a623643 `YoutubePlaylistLinkHandlerFactoryTest`: Use parameterized tests 2022-08-14 14:48:31 +02:00
litetex
938e69a16a Fixed `YoutubePlaylistLinkHandlerFactoryTest` 2022-08-14 14:48:30 +02:00
litetex
844de3e378 Fix `YoutubeStreamLinkHandlerFactoryTest ` and parameterized the tests 
used code from #833
 2022-08-14 14:48:30 +02:00
litetex
c1a72b8240 Use Android Studios code style 2022-08-14 14:48:29 +02:00
litetex
fc27b8a5b8 Use preexisting `ContentNotAvailableException` 2022-08-14 14:48:29 +02:00
Stypox
17bad6cedf Fix test about exception type: NPE, not IllegalArgment 
The relevant change was made in #877
 2022-08-14 14:48:28 +02:00
litetex
ecfc370685 Fixed all YTMixPlaylists 
Added option to choose if you want to consent or not - currently this is done by a static variable in ``YoutubeParsingHelper`` - may not be the best long-term solution but for now the tests work again (in EU countries) 🥳
 2022-08-14 14:48:27 +02:00
litetex
2b6fe294b2 Fixed `YoutubeMusicSearchExtractorTest` 
The renderers and queries got changed.
 2022-08-14 14:48:27 +02:00
litetex
d6586da614 Failing test works locally without any problems. Reformatted it a bit. 2022-08-14 14:48:27 +02:00
litetex
bf3ae5e679 Fixed `SoundcloudStreamLinkHandlerFactoryTest` 
* Removed the unnecessary ``public``
* Use parameterized tests
* One song got removed (which caused the test failure), replaced it with another song
 2022-08-14 14:48:27 +02:00
litetex
504f81036e Fixed `PeertubePlaylistExtractorTest` 
Also removed the unnecessary ``public`` and reformatted the code a bit
 2022-08-14 14:48:26 +02:00
litetex
e7c12258f4 Fixed `BandcampSearchExtractorTest` 
Also removed the unnecessary ``public``
 2022-08-14 14:48:26 +02:00
litetex
9e8724df4d Fixed `YoutubeStreamExtractorLivestreamTest` 
The "Lofi Girl"-stream got interrupted by a copyright strike and had to be restarted. Because of this a new id is now used.
 2022-08-14 14:48:26 +02:00
Tobi
76aad92fa5
Merge pull request #890 from AudricV/yt-clients-update-improvements-and-extraction-fixes 
[YouTube] Update clients version and improve extraction of API key and client version of WEB client
 2022-08-12 23:32:41 +02:00
AudricV
472f5d9e9c
[YouTube] Update mocks 2022-08-12 19:20:32 +02:00
AudricV
7bdca33a87
[YouTube] Ensure that an additional player response is the correct one 
If YouTube detect that requests come from a third party client, they may
replace the real player response by another one of a video saying that this
content is not available on this app and to watch it on the latest version of
YouTube. We can detect this by checking whether the video ID of the player
response returned is the same as the one requested by the extractor.
 2022-08-12 19:20:31 +02:00
AudricV
c82317e318
[YouTube] Spoof more mobile clients 
Additional parameters have been added to the player requests of ANDROID and IOS
clients:

- for both clients: osName and osVersion: their respective values are:
  - for the ANDROID one: Android and 12;
  - for the IOS one: iOS and 15.6.0.19G71.
- for the ANDROID client: androidTargetSdkVersion, with the Android SDK version
  corresponding to the Android version used in the player requests of this
  client. This parameter is now required with this client to be sure to get a
  correct player response, otherwise, the one of a video saying that this
  content is not available in this app and to watch it with the latest version
  of YouTube can be returned instead;
- for the IOS client: deviceMake, with Apple as its value.

The iOS version sent in the IOS client player requests has been also updated to
the version 15.6 of the OS.

Finally, a comment about the requirement to use the signature timestamp from
the player JavaScript base file for HTML5 player requests on videos with
obfuscated URLs has been added and replaces a previous one which may be not
true.
 2022-08-12 19:20:31 +02:00
AudricV
d0549a5a52
[YouTube] Update client versions and use a real version for the iOS client 
The iOS version can be got easily in fact, by looking at the What's New section of the App Store' app page.
 2022-08-12 19:20:31 +02:00
AudricV
d7e678aca2
[YouTube] Improve WEB client version and API key HTML extraction 
Common code in WEB client version HTML extraction has been deduplicated, usage of the Java 8 Stream API has been made and initial data fallback has been used as a last resort.
This means that the client version extraction from regexes will be used before this fallback, as it doesn't contain the full client version.
This can be used as a way to fingerprint the extractor, even if it seems to be not the case.
 2022-08-12 19:20:30 +02:00
AudricV
6a885ef5ab
Merge pull request #891 from Theta-Dev/fix-throttling-decrypter 
[YouTube] Fix extraction of more complex nsig functions
 2022-08-12 18:24:21 +02:00
AudricV
5b548340e8
[YouTube] Catch any exception in YoutubeThrottlingDecrypter.apply and improve docs 
This will prevent any future extractor break due to decryption failure, like it was excepted to be the case before.

Some documentation about the throttling decryption has been also improved.
 2022-08-12 17:49:36 +02:00
ThetaDev
52ded6e3d7
Handle curly braces inside strings in StringUtils.matchToClosingParenthesis 
This is required to extract fully more complex YouTube nsig functions.
 2022-08-12 16:32:00 +02:00
Stypox
d12003651b
Merge pull request #886 from Isira-Seneviratne/toArray_improvements 
Make improvements to methods using toArray().
 2022-08-06 22:34:23 +02:00
Isira Seneviratne
7daca10a06 Make improvements to methods using toArray(). 2022-08-06 05:21:12 +05:30
Stypox
2906be22af
Merge pull request #881 from Isira-Seneviratne/String_join 
Use String.join() and Collectors.joining().
 2022-08-04 12:09:33 +02:00
Stypox
4ddb96a86f
Merge pull request #883 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.1 
Bump gson from 2.9.0 to 2.9.1
 2022-08-04 11:20:41 +02:00
Isira Seneviratne
64771c5712 Use String.join() and Collectors.joining(). 2022-08-04 05:18:13 +05:30
Stypox
fc8b5ebbc6
Merge pull request #878 from Isira-Seneviratne/Use_Collections 
Use Collections methods.
 2022-08-03 22:50:41 +02:00
Stypox
4a4939d89c
Merge pull request #877 from Isira-Seneviratne/Use_Objects_requireNonNull 
Use Objects.requireNonNull().
 2022-08-03 22:45:35 +02:00
Stypox
c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0 
Bump junit-bom from 5.8.2 to 5.9.0
 2022-08-03 21:06:20 +02:00
dependabot[bot]
325af31e5f
Bump gson from 2.9.0 to 2.9.1 
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-08-01 09:26:14 +00:00
dependabot[bot]
d905636021
Bump junit-bom from 5.8.2 to 5.9.0 
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-27 09:20:12 +00:00
Isira Seneviratne
1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne
ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne
682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox
8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions 
Use minimum required permissions for GitHub workflows
 2022-07-13 18:58:32 +02:00
Stypox
954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2 
Bump jsoup from 1.15.1 to 1.15.2
 2022-07-13 17:48:06 +02:00
Mohammed Anas
f57049d2c0
Use temurin instead of adopt (#868) 
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
 2022-07-10 18:38:08 +03:00
Stypox
5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1 
Bump spotbugs-annotations from 4.7.0 to 4.7.1
 2022-07-06 14:23:44 +02:00
Stypox
5ab74b3631
Merge pull request #857 from FireMasterK/video-title 
Get original untranslated title for YouTube
 2022-07-06 10:26:45 +02:00
dependabot[bot]
122365005a
Bump jsoup from 1.15.1 to 1.15.2 
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-07-04 09:38:16 +00:00
mhmdanas
414186cff5 Use minimum required permissions for GitHub workflows 
This reduces the attack surface if the workflows are ever compromised.
 2022-07-03 23:55:28 +03:00