The regular expression pattern used to match tags in URLs in the DanbooruRipper class's getTag method has been updated. Handles both /posts?tags=rakeemspoon&z=2 and /posts?tags=rakeemspoon
([^&]*) -> anything but &
(?:&z=([0-9]+))? -> optional regex -> if & exists, then we check z=numbers
A new implementation line was added to the build.gradle.kts file for OkHttp library. This change is needed in order to extend the project's capabilities for HTTP network requests for ones with encryption/compression.
Replaced the existing request method in the DanbooruRipper class with OkHttpClient for better performance. Implemented error handling and response checking to ensure the http connection's success. The request headers were also updated to optimize and secure the connection.
In addition, connection was able to work via
Connection connection = Jsoup.connect("https://danbooru.donmai.us/posts?page=1&tags=rakeemspoon")
.ignoreContentType(true)
.userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8")
.header("Accept-Language", "en-US,en;q=0.5")
.header("Upgrade-Insecure-Requests", "1")
.header("Sec-Fetch-Dest", "document")
.header("Sec-Fetch-Mode", "navigate")
.header("Sec-Fetch-Site", "same-origin")
.header("Sec-Fetch-User", "?1")
.header("Pragma", "no-cache")
.header("Cache-Control", "no-cache")
.header("Referer", "https://danbooru.donmai.us/posts?page=4&tags=rakeemspoon")
.header("Accept-Encoding", "gzip, deflate, br");
However, the returned data was gibberish - I had no functional idea on how to fix it.
