Commit graph

1322 commits

Author SHA1 Message Date
TiA4f8R
890cbba625
Update PaidContent exception 2021-03-05 16:38:51 +01:00
TobiGr
da3cfa967d
Handle age-restricted videos 2021-03-05 16:38:50 +01:00
TiA4f8R
448b68700c
Use final in exceptions and add AgeRestrictedException
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available. 
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R
f15d7837a1
Fix SoundCloud tests 2021-03-05 16:38:45 +01:00
TiA4f8R
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test 2021-03-05 16:38:43 +01:00
TiA4f8R
59d6d3f04e
Fix YouTube tests 2021-03-05 16:38:42 +01:00
TiA4f8R
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException 2021-03-05 16:38:40 +01:00
TiA4f8R
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr
cb07ffa1eb Release 0.20.11 2021-03-05 15:35:11 +01:00
Tobi
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
ed850d0688 [youtube] improve comments extraction performance
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
   -> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr
440a808b8a Update mocks for channels 2021-03-05 13:13:36 +01:00
Tobi
b608587e4d
Merge pull request #572 from golfinq/fixed-comments
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq
05213175cd
minor edit, made variable final 2021-03-05 02:49:15 -05:00
golfinq
8bceb57a6c
formatting fix 2021-03-05 02:46:06 -05:00
golfinq
ae48bdea4c
Updated to reflect variable name change 2021-03-05 02:40:36 -05:00
TiA4f8R
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
Because it's unused with the new request.

Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr
2b671b15ce [YouTube] Use new continuation API for channels
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr
33594d4aff [SoundCloud] Fix client id extraction
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr
fb2f37d223 [SoundCloud] Update hardcoded client id 2021-03-04 21:25:41 +01:00
XiangRongLin
9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin
506cc5fbe9 regenerate mocks 2021-03-04 19:01:58 +01:00
XiangRongLin
03b00ff1d6 Move toJsonArray and toJsonObject to JsonUtils 2021-03-04 18:58:51 +01:00
FireMasterK
668b080b3a Use dynamic key. 2021-03-04 08:45:14 +01:00
XiangRongLin
43b46bd408 Use new youtube continuations api for playlists
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.

quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin
beb05bd05c Replace test url, because playlist was deleted 2021-03-03 19:46:14 +01:00
Tobi
5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi
965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R
b2cf41496d
Set extractor encoding to UTF-8
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
a9aa385bb3
Merge pull request #554 from B0pol/test-verified
use DownloaderFactory instead of DownloaderTestImpl for search Channe…
2021-02-20 12:12:43 +01:00
0c831afa7b use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified 2021-02-20 11:41:30 +01:00
27a20e41cd
Merge pull request #550 from FireMasterK/add-more-data
Add more data.
2021-02-18 20:28:27 +01:00
FireMasterK
5333d8a98b
Add more data. 2021-02-19 00:55:03 +05:30
FireMasterK
88890d81a5
Update gradle wrapper. 2021-02-18 21:15:12 +05:30
9c1a7f7df6
Merge pull request #521 from B0pol/verified
Add uploader verified by service extraction
2021-02-18 16:39:53 +01:00
1a322ad8ed Add uploader verified by service extraction 2021-02-18 16:36:40 +01:00
db253e202b
Merge pull request #535 from TeamPiped/adaptive-parsing
Adaptive Stream parsing
2021-02-18 15:55:27 +01:00
FireMasterK
c24afa2cbb Permalink URL. 2021-02-18 20:16:53 +05:30
FireMasterK
8d54401233 Add fps field as well. 2021-02-18 20:16:53 +05:30
FireMasterK
4d096be14a Revert "Update gradle wrapper."
This reverts commit 591856e3a11998c4b8ba87b5e361e2b9556d760f.
2021-02-18 20:16:53 +05:30
FireMasterK
5b70645d85 Add data to muxed streams aswell. 2021-02-18 20:16:53 +05:30
FireMasterK
e975d33fbe Change jdoc / comments. 2021-02-18 20:16:53 +05:30
FireMasterK
f31b2a68fd use getters/setters. 2021-02-18 20:16:53 +05:30
FireMasterK
d4945ac55c Constructor cleanup. 2021-02-18 20:16:52 +05:30