Commit graph

1798 commits

Author SHA1 Message Date
TiA4f8R
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
TiA4f8R
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
TiA4f8R
26f1b4e7dc
Support SoundCloud HLS by using a workaround
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
Tobi
def745b801
Merge pull request #579 from fynngodau/bandcamp-fix-linkhandler
[Bandcamp] Fix link handler acceptance behaviour
2021-03-14 12:56:10 +01:00
Fynn Godau
2e57a8f24f [Bandcamp] Fix link handler acceptance behaviour
* Test for bandcamp footer instead of meta tag (which is not present on
  all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
TobiGr
70d9e389b9 [Bandcamp] Add tests for external bandcamp artists
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
TobiGr
f9d06252f2 [Bandcamp] Fix tests 2021-03-13 17:42:39 +01:00
Tobi
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
Tobi
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
Unignore more tests
2021-03-07 13:15:24 +01:00
Tobi
021da75f24
Merge pull request #232 from fynngodau/dev
Bandcamp support
2021-03-05 21:37:57 +01:00
TobiGr
22fa131922 Merge branch 'dev' into bandcamp 2021-03-05 19:45:37 +01:00
TiA4f8R
e55284bb8f
Code style improvements
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
TobiGr
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase 2021-03-05 16:38:52 +01:00
TiA4f8R
890cbba625
Update PaidContent exception 2021-03-05 16:38:51 +01:00
TobiGr
da3cfa967d
Handle age-restricted videos 2021-03-05 16:38:50 +01:00
TiA4f8R
448b68700c
Use final in exceptions and add AgeRestrictedException
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
TobiGr
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available. 
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
TobiGr
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
TiA4f8R
f15d7837a1
Fix SoundCloud tests 2021-03-05 16:38:45 +01:00
TiA4f8R
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test 2021-03-05 16:38:43 +01:00
TiA4f8R
59d6d3f04e
Fix YouTube tests 2021-03-05 16:38:42 +01:00
TiA4f8R
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException 2021-03-05 16:38:40 +01:00
TiA4f8R
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
TiA4f8R
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
TobiGr
cb07ffa1eb Release 0.20.11 2021-03-05 15:35:11 +01:00
Tobi
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
Tobi
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
Tobi
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
ed850d0688 [youtube] improve comments extraction performance
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
   -> parse responseBody less times
2021-03-05 13:37:08 +01:00
TobiGr
440a808b8a Update mocks for channels 2021-03-05 13:13:36 +01:00
Tobi
b608587e4d
Merge pull request #572 from golfinq/fixed-comments
Fixes missing comments under videos
2021-03-05 11:42:40 +01:00
golfinq
05213175cd
minor edit, made variable final 2021-03-05 02:49:15 -05:00
golfinq
8bceb57a6c
formatting fix 2021-03-05 02:46:06 -05:00
golfinq
ae48bdea4c
Updated to reflect variable name change 2021-03-05 02:40:36 -05:00
TiA4f8R
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
Because it's unused with the new request.

Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
TobiGr
2b671b15ce [YouTube] Use new continuation API for channels
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
TobiGr
33594d4aff [SoundCloud] Fix client id extraction
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
TobiGr
fb2f37d223 [SoundCloud] Update hardcoded client id 2021-03-04 21:25:41 +01:00
XiangRongLin
9256b3b848
Merge pull request #567 from XiangRongLin/playlist_continuations
Playlist continuations
2021-03-04 19:05:32 +01:00
XiangRongLin
506cc5fbe9 regenerate mocks 2021-03-04 19:01:58 +01:00
XiangRongLin
03b00ff1d6 Move toJsonArray and toJsonObject to JsonUtils 2021-03-04 18:58:51 +01:00
FireMasterK
668b080b3a Use dynamic key. 2021-03-04 08:45:14 +01:00
XiangRongLin
43b46bd408 Use new youtube continuations api for playlists
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.

quick and dirty solution.
2021-03-03 19:49:26 +01:00
XiangRongLin
beb05bd05c Replace test url, because playlist was deleted 2021-03-03 19:46:14 +01:00
XiangRongLin
e13e237392 Fix typo and reword some explanations 2021-02-26 17:48:03 +01:00
Tobi
5d594cfded
Merge pull request #549 from FireMasterK/gradle-update
Update gradle wrapper.
2021-02-24 15:59:28 +01:00
Tobi
965a66bc87
Merge pull request #557 from TiA4f8R/fix-encoding-errors-windows
Set extractor encoding to UTF-8
2021-02-24 15:49:58 +01:00
TiA4f8R
b2cf41496d
Set extractor encoding to UTF-8
When compiling the extractor on Windows, you get more than 100 errors with the message error: unmappable character for encoding Cp1252, especially in timeago-parser tasks. This commit fixes it by setting the encoding for all compileJava and compileTestJava tasks to UTF-8.
2021-02-23 14:38:20 +01:00
TobiGr
a1688fe953 Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils 2021-02-21 13:51:12 +01:00