Merge branch 'master' into source/neox
commit
089191d6f1
@ -0,0 +1,91 @@
|
|||||||
|
# Contributing
|
||||||
|
|
||||||
|
The following is guide for creating a Kotatsu parsers. Thanks for taking the time to contribute!
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
Before you start, please note that the ability to use following technologies is **required**.
|
||||||
|
|
||||||
|
- Basic [Android development](https://developer.android.com/)
|
||||||
|
- [Kotlin](https://kotlinlang.org/)
|
||||||
|
- Web scraping ([JSoup](https://jsoup.org/)) or JSON API
|
||||||
|
|
||||||
|
### Tools
|
||||||
|
|
||||||
|
- [Android Studio](https://developer.android.com/studio)
|
||||||
|
- [IntelliJ IDEA](https://www.jetbrains.com/idea/) (Community edition is enough)
|
||||||
|
- Android device (or emulator)
|
||||||
|
|
||||||
|
Kotatsu parsers is not a part of Android application, but you can easily develop and test it directly inside an Android
|
||||||
|
application project and relocate it to the library project when done.
|
||||||
|
|
||||||
|
### Before you start
|
||||||
|
|
||||||
|
First, take a look at `kotatsu-parsers` project structure. Each parser is a single class that
|
||||||
|
extends `MangaParser` class and have a `MangaSourceParser` annotation.
|
||||||
|
Also pay attention on extensions in `util` package. For example, extensions from `Jsoup` file
|
||||||
|
should be used instead of existing JSoup functions because they have better nullability support
|
||||||
|
and improved error messages.
|
||||||
|
|
||||||
|
## Writing your parser
|
||||||
|
|
||||||
|
So, you want to create a parser, that will provide access to manga from a website.
|
||||||
|
First, you should explore a website for API availability.
|
||||||
|
If it does not contain any documentation about
|
||||||
|
API, [explore network requests](https://firefox-source-docs.mozilla.org/devtools-user/):
|
||||||
|
some websites use ajax.
|
||||||
|
|
||||||
|
- [Example](https://github.com/KotatsuApp/kotatsu-parsers/blob/master/src/main/kotlin/org/koitharu/kotatsu/parsers/site/DesuMeParser.kt)
|
||||||
|
of Json API usage.
|
||||||
|
- [Example](https://github.com/KotatsuApp/kotatsu-parsers/blob/master/src/main/kotlin/org/koitharu/kotatsu/parsers/site/AnibelParser.kt)
|
||||||
|
of GraphQL API usage
|
||||||
|
- [Example](https://github.com/KotatsuApp/kotatsu-parsers/blob/master/src/main/kotlin/org/koitharu/kotatsu/parsers/site/MangaTownParser.kt)
|
||||||
|
of pure HTML parsing.
|
||||||
|
|
||||||
|
If website is based on some engine it is rationally to use common base class for this one (for example, Madara wordress
|
||||||
|
theme
|
||||||
|
and the `MadaraParser` class)
|
||||||
|
|
||||||
|
### Parser class skeleton
|
||||||
|
|
||||||
|
Parser class must have exactly one primary constructor parameter of type `MangaLoaderContext` and have an
|
||||||
|
`MangaSourceParser` annotation that provides internal name, title and language of a manga source.
|
||||||
|
|
||||||
|
All functions in `MangaParser` class are documented. Pay attention to some peculiarities:
|
||||||
|
|
||||||
|
- Never hardcode domain. Specify default domain in `configKeyDomain` field and obtain an actual one using `getDomain()`.
|
||||||
|
- All ids must be unique and domain-independent. Use `generateUid` functions with relative url or some internal id which
|
||||||
|
is unique across the manga source.
|
||||||
|
- `sortOrders` set should not be empty. If your source is not support sorting, specify one most relevance value.
|
||||||
|
- If you cannot obtain direct links to pages images inside `getPages` method, it is ok to use an intermediate url
|
||||||
|
as `Page.url` and fetch a direct link at `getPageUrl` function.
|
||||||
|
- You can use _asserts_ to check some optional fields. For example. `Manga.author` field is not required, but if your
|
||||||
|
source provide such information, add `assert(it != null)`. This will not have any effect on production but help to
|
||||||
|
find issues during unit testing.
|
||||||
|
- If your source website (or it's api) uses pages for pagination instead of offset you should extend `PagedMangaParser`
|
||||||
|
instead of `MangaParser`.
|
||||||
|
- Your parser may also implement the `Interceptor` interface for additional manipulation of all network requests and/or
|
||||||
|
responses, including image loading.
|
||||||
|
|
||||||
|
## Development process
|
||||||
|
|
||||||
|
During the development it is recommended (but not necessary) to write it directly
|
||||||
|
in the Kotatsu android application project. You can use `core.parser.DummyParser` class as a sandbox. `Dummy` manga
|
||||||
|
source is available in debug Kotatsu build.
|
||||||
|
|
||||||
|
Once parser is ready you can relocate your code into `kotatsu-parsers` library project in a `site` package and create a
|
||||||
|
Pull Request.
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
It is recommended to run unit tests before submitting a PR.
|
||||||
|
|
||||||
|
- Temporary modify the `MangaSources` annotation class: specify your parser(s) name(s) and change mode
|
||||||
|
to `EnumSource.Mode.INCLUDE`
|
||||||
|
- Run the `MangaParserTest` (`gradlew :test --tests "org.koitharu.kotatsu.parsers.MangaParserTest"`)
|
||||||
|
- Optionally, you can run the `generateTestsReport` gradle task to get a pretty readable html report from test results.
|
||||||
|
|
||||||
|
## Help
|
||||||
|
|
||||||
|
If you need a help or have some questions, ask a community in our [Telegram chat](https://t.me/kotatsuapp)
|
||||||
|
or [Discord server](https://discord.gg/NNJ5RgVBC5).
|
||||||
@ -1,13 +1,21 @@
|
|||||||
package org.koitharu.kotatsu.parsers.config
|
package org.koitharu.kotatsu.parsers.config
|
||||||
|
|
||||||
sealed class ConfigKey<T>(
|
sealed class ConfigKey<T>(
|
||||||
val key: String,
|
@JvmField val key: String,
|
||||||
) {
|
) {
|
||||||
|
|
||||||
abstract val defaultValue: T
|
abstract val defaultValue: T
|
||||||
|
|
||||||
class Domain(
|
class Domain(
|
||||||
override val defaultValue: String,
|
override val defaultValue: String,
|
||||||
val presetValues: Array<String>?,
|
@JvmField val presetValues: Array<String>?,
|
||||||
) : ConfigKey<String>("domain")
|
) : ConfigKey<String>("domain")
|
||||||
|
|
||||||
|
class ShowSuspiciousContent(
|
||||||
|
override val defaultValue: Boolean,
|
||||||
|
) : ConfigKey<Boolean>("show_suspicious")
|
||||||
|
|
||||||
|
class UserAgent(
|
||||||
|
override val defaultValue: String,
|
||||||
|
) : ConfigKey<String>("user_agent")
|
||||||
}
|
}
|
||||||
@ -1,7 +1,9 @@
|
|||||||
package org.koitharu.kotatsu.parsers.exception
|
package org.koitharu.kotatsu.parsers.exception
|
||||||
|
|
||||||
|
import okhttp3.Headers
|
||||||
import okio.IOException
|
import okio.IOException
|
||||||
|
|
||||||
class CloudFlareProtectedException(
|
class CloudFlareProtectedException(
|
||||||
val url: String,
|
val url: String,
|
||||||
|
val headers: Headers,
|
||||||
) : IOException("Protected by CloudFlare: $url")
|
) : IOException("Protected by CloudFlare: $url")
|
||||||
@ -0,0 +1,118 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.network
|
||||||
|
|
||||||
|
import okhttp3.*
|
||||||
|
import okhttp3.MediaType.Companion.toMediaType
|
||||||
|
import okhttp3.RequestBody.Companion.toRequestBody
|
||||||
|
import org.json.JSONObject
|
||||||
|
import org.jsoup.HttpStatusException
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.AuthRequiredException
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.GraphQLException
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.NotFoundException
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaSource
|
||||||
|
import org.koitharu.kotatsu.parsers.util.await
|
||||||
|
import org.koitharu.kotatsu.parsers.util.parseJson
|
||||||
|
import java.net.HttpURLConnection
|
||||||
|
|
||||||
|
class OkHttpWebClient(
|
||||||
|
private val httpClient: OkHttpClient,
|
||||||
|
private val mangaSource: MangaSource,
|
||||||
|
) : WebClient {
|
||||||
|
|
||||||
|
override suspend fun httpGet(url: HttpUrl, extraHeaders: Headers?): Response {
|
||||||
|
val request = Request.Builder()
|
||||||
|
.get()
|
||||||
|
.url(url)
|
||||||
|
.addTags()
|
||||||
|
if (extraHeaders != null) {
|
||||||
|
request.headers(extraHeaders)
|
||||||
|
}
|
||||||
|
return httpClient.newCall(request.build()).await().ensureSuccess()
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun httpHead(url: HttpUrl): Response {
|
||||||
|
val request = Request.Builder()
|
||||||
|
.head()
|
||||||
|
.url(url)
|
||||||
|
.addTags()
|
||||||
|
return httpClient.newCall(request.build()).await().ensureSuccess()
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun httpPost(url: HttpUrl, form: Map<String, String>): Response {
|
||||||
|
val body = FormBody.Builder()
|
||||||
|
form.forEach { (k, v) ->
|
||||||
|
body.addEncoded(k, v)
|
||||||
|
}
|
||||||
|
val request = Request.Builder()
|
||||||
|
.post(body.build())
|
||||||
|
.url(url)
|
||||||
|
.addTags()
|
||||||
|
return httpClient.newCall(request.build()).await().ensureSuccess()
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun httpPost(url: HttpUrl, payload: String): Response {
|
||||||
|
val body = FormBody.Builder()
|
||||||
|
payload.split('&').forEach {
|
||||||
|
val pos = it.indexOf('=')
|
||||||
|
if (pos != -1) {
|
||||||
|
val k = it.substring(0, pos)
|
||||||
|
val v = it.substring(pos + 1)
|
||||||
|
body.addEncoded(k, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
val request = Request.Builder()
|
||||||
|
.post(body.build())
|
||||||
|
.url(url)
|
||||||
|
.addTags()
|
||||||
|
return httpClient.newCall(request.build()).await().ensureSuccess()
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun httpPost(url: HttpUrl, body: JSONObject): Response {
|
||||||
|
val mediaType = "application/json; charset=utf-8".toMediaType()
|
||||||
|
val requestBody = body.toString().toRequestBody(mediaType)
|
||||||
|
val request = Request.Builder()
|
||||||
|
.post(requestBody)
|
||||||
|
.url(url)
|
||||||
|
.addTags()
|
||||||
|
return httpClient.newCall(request.build()).await().ensureSuccess()
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun graphQLQuery(endpoint: String, query: String): JSONObject {
|
||||||
|
val body = JSONObject()
|
||||||
|
body.put("operationName", null as Any?)
|
||||||
|
body.put("variables", JSONObject())
|
||||||
|
body.put("query", "{$query}")
|
||||||
|
val json = httpPost(endpoint, body).parseJson()
|
||||||
|
json.optJSONArray("errors")?.let {
|
||||||
|
if (it.length() != 0) {
|
||||||
|
throw GraphQLException(it)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return json
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun Request.Builder.addTags(): Request.Builder {
|
||||||
|
tag(MangaSource::class.java, mangaSource)
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun Response.ensureSuccess(): Response {
|
||||||
|
val exception: Exception? = when (code) { // Catch some error codes, not all
|
||||||
|
HttpURLConnection.HTTP_NOT_FOUND -> NotFoundException(message, request.url.toString())
|
||||||
|
HttpURLConnection.HTTP_UNAUTHORIZED -> request.tag(MangaSource::class.java)?.let {
|
||||||
|
AuthRequiredException(it)
|
||||||
|
} ?: HttpStatusException(message, code, request.url.toString())
|
||||||
|
|
||||||
|
in 400..599 -> HttpStatusException(message, code, request.url.toString())
|
||||||
|
else -> null
|
||||||
|
}
|
||||||
|
if (exception != null) {
|
||||||
|
runCatching {
|
||||||
|
close()
|
||||||
|
}.onFailure {
|
||||||
|
exception.addSuppressed(it)
|
||||||
|
}
|
||||||
|
throw exception
|
||||||
|
}
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,87 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.network
|
||||||
|
|
||||||
|
import okhttp3.Headers
|
||||||
|
import okhttp3.HttpUrl
|
||||||
|
import okhttp3.HttpUrl.Companion.toHttpUrl
|
||||||
|
import okhttp3.Response
|
||||||
|
import org.json.JSONObject
|
||||||
|
|
||||||
|
interface WebClient {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a GET http request to specific url
|
||||||
|
* @param url
|
||||||
|
*/
|
||||||
|
suspend fun httpGet(url: String): Response = httpGet(url.toHttpUrl())
|
||||||
|
|
||||||
|
suspend fun httpGet(url: String, extraHeaders: Headers?): Response = httpGet(url.toHttpUrl(), extraHeaders)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a GET http request to specific url
|
||||||
|
* @param url
|
||||||
|
*/
|
||||||
|
suspend fun httpGet(url: HttpUrl): Response = httpGet(url, null)
|
||||||
|
|
||||||
|
suspend fun httpGet(url: HttpUrl, extraHeaders: Headers?): Response
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a HEAD http request to specific url
|
||||||
|
* @param url
|
||||||
|
*/
|
||||||
|
suspend fun httpHead(url: String): Response = httpHead(url.toHttpUrl())
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a HEAD http request to specific url
|
||||||
|
* @param url
|
||||||
|
*/
|
||||||
|
suspend fun httpHead(url: HttpUrl): Response
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a POST http request to specific url with `multipart/form-data` payload
|
||||||
|
* @param url
|
||||||
|
* @param form payload as key=>value map
|
||||||
|
*/
|
||||||
|
suspend fun httpPost(url: String, form: Map<String, String>): Response = httpPost(url.toHttpUrl(), form)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a POST http request to specific url with `multipart/form-data` payload
|
||||||
|
* @param url
|
||||||
|
* @param form payload as key=>value map
|
||||||
|
*/
|
||||||
|
suspend fun httpPost(url: HttpUrl, form: Map<String, String>): Response
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a POST http request to specific url with `multipart/form-data` payload
|
||||||
|
* @param url
|
||||||
|
* @param payload payload as `key=value` string with `&` separator
|
||||||
|
*/
|
||||||
|
suspend fun httpPost(url: String, payload: String): Response = httpPost(url.toHttpUrl(), payload)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a POST http request to specific url with `multipart/form-data` payload
|
||||||
|
* @param url
|
||||||
|
* @param payload payload as `key=value` string with `&` separator
|
||||||
|
*/
|
||||||
|
suspend fun httpPost(url: HttpUrl, payload: String): Response
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a POST http request to specific url with json payload
|
||||||
|
* @param url
|
||||||
|
* @param body
|
||||||
|
*/
|
||||||
|
suspend fun httpPost(url: String, body: JSONObject): Response = httpPost(url.toHttpUrl(), body)
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a POST http request to specific url with json payload
|
||||||
|
* @param url
|
||||||
|
* @param body
|
||||||
|
*/
|
||||||
|
suspend fun httpPost(url: HttpUrl, body: JSONObject): Response
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do a GraphQL request to specific url
|
||||||
|
* @param endpoint an url
|
||||||
|
* @param query GraphQL request payload
|
||||||
|
*/
|
||||||
|
suspend fun graphQLQuery(endpoint: String, query: String): JSONObject
|
||||||
|
}
|
||||||
@ -0,0 +1,240 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site
|
||||||
|
|
||||||
|
import kotlinx.coroutines.async
|
||||||
|
import kotlinx.coroutines.awaitAll
|
||||||
|
import kotlinx.coroutines.coroutineScope
|
||||||
|
import okhttp3.Headers
|
||||||
|
import org.jsoup.nodes.Element
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.PagedMangaParser
|
||||||
|
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.ParseException
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.json.getIntOrDefault
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("BENTOMANGA", "Bentomanga", "fr")
|
||||||
|
internal class BentomangaParser(context: MangaLoaderContext) : PagedMangaParser(context, MangaSource.BENTOMANGA, 10) {
|
||||||
|
|
||||||
|
override val sortOrders: Set<SortOrder> = EnumSet.of(
|
||||||
|
SortOrder.UPDATED,
|
||||||
|
SortOrder.POPULARITY,
|
||||||
|
SortOrder.RATING,
|
||||||
|
SortOrder.NEWEST,
|
||||||
|
SortOrder.ALPHABETICAL,
|
||||||
|
)
|
||||||
|
|
||||||
|
override val configKeyDomain = ConfigKey.Domain("www.bentomanga.com", null)
|
||||||
|
|
||||||
|
override val headers: Headers = Headers.Builder()
|
||||||
|
.add("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0")
|
||||||
|
.build()
|
||||||
|
|
||||||
|
init {
|
||||||
|
paginator.firstPage = 0
|
||||||
|
searchPaginator.firstPage = 0
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getListPage(
|
||||||
|
page: Int,
|
||||||
|
query: String?,
|
||||||
|
tags: Set<MangaTag>?,
|
||||||
|
sortOrder: SortOrder,
|
||||||
|
): List<Manga> {
|
||||||
|
val url = urlBuilder()
|
||||||
|
.addPathSegment("manga_list")
|
||||||
|
.addQueryParameter("limit", page.toString())
|
||||||
|
.addQueryParameter(
|
||||||
|
"order_by",
|
||||||
|
when (sortOrder) {
|
||||||
|
SortOrder.UPDATED -> "update"
|
||||||
|
SortOrder.POPULARITY -> "views"
|
||||||
|
SortOrder.RATING -> "top"
|
||||||
|
SortOrder.NEWEST -> "create"
|
||||||
|
SortOrder.ALPHABETICAL -> "name"
|
||||||
|
},
|
||||||
|
)
|
||||||
|
if (!tags.isNullOrEmpty()) {
|
||||||
|
url.addQueryParameter("withCategories", tags.joinToString(",") { it.key })
|
||||||
|
}
|
||||||
|
if (!query.isNullOrEmpty()) {
|
||||||
|
url.addQueryParameter("search", query)
|
||||||
|
}
|
||||||
|
val root = webClient.httpGet(url.build()).parseHtml().requireElementById("mangas_content")
|
||||||
|
return root.select(".manga[data-manga]").map { div ->
|
||||||
|
val header = div.selectFirstOrThrow(".manga_header")
|
||||||
|
val href = header.selectFirstOrThrow("a").attrAsRelativeUrl("href")
|
||||||
|
Manga(
|
||||||
|
id = generateUid(href),
|
||||||
|
title = div.selectFirstOrThrow("h1").text(),
|
||||||
|
altTitle = null,
|
||||||
|
url = href,
|
||||||
|
publicUrl = href.toAbsoluteUrl(domain),
|
||||||
|
rating = div.getElementsByAttributeValue("data-icon", "avg_rate")
|
||||||
|
.firstOrNull()
|
||||||
|
?.textOrNull()
|
||||||
|
?.toFloatOrNull()
|
||||||
|
?.div(10f)
|
||||||
|
?: RATING_UNKNOWN,
|
||||||
|
isNsfw = div.selectFirst(".badge-adult_content") != null,
|
||||||
|
coverUrl = div.selectFirstOrThrow("img").src(),
|
||||||
|
tags = div.selectFirst(".component-manga-categories")
|
||||||
|
.assertNotNull("tags")
|
||||||
|
?.select("a")
|
||||||
|
?.mapToSet { a ->
|
||||||
|
MangaTag(
|
||||||
|
title = a.text().toTitleCase(sourceLocale),
|
||||||
|
key = a.attr("href").substringAfterLast('='),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}.orEmpty(),
|
||||||
|
state = null,
|
||||||
|
author = null,
|
||||||
|
description = div.selectFirst(".manga_synopsis")?.html().assertNotNull("description"),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga {
|
||||||
|
val mangaUrl = manga.url.toAbsoluteUrl(domain)
|
||||||
|
val root = webClient.httpGet(mangaUrl).parseHtml()
|
||||||
|
.requireElementById("container_manga_show")
|
||||||
|
return manga.copy(
|
||||||
|
altTitle = root.selectFirst(".component-manga-title_alt")?.textOrNull().assertNotNull("altTitle"),
|
||||||
|
description = root.selectFirst(".datas_synopsis")?.html().assertNotNull("description")
|
||||||
|
?: manga.description,
|
||||||
|
state = when (root.selectFirst(".datas_more-status-data")?.textOrNull().assertNotNull("status")) {
|
||||||
|
"En cours" -> MangaState.ONGOING
|
||||||
|
else -> null
|
||||||
|
},
|
||||||
|
author = root.selectFirst(".datas_more-authors-people")?.textOrNull().assertNotNull("author"),
|
||||||
|
chapters = run {
|
||||||
|
val input = root.selectFirst("input[name=\"limit\"]") ?: return@run parseChapters(root)
|
||||||
|
val max = input.attr("max").toInt()
|
||||||
|
if (max <= 1) {
|
||||||
|
parseChapters(root)
|
||||||
|
} else {
|
||||||
|
coroutineScope {
|
||||||
|
val result = ArrayList<MangaChapter>(parseChapters(root))
|
||||||
|
result.ensureCapacity(result.size * max)
|
||||||
|
(2..max).map { i ->
|
||||||
|
async {
|
||||||
|
loadChapters(mangaUrl, i)
|
||||||
|
}
|
||||||
|
}.awaitAll()
|
||||||
|
.flattenTo(result)
|
||||||
|
result
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}.reversed(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
val chapterUrl = chapter.url.toAbsoluteUrl(domain)
|
||||||
|
val doc = webClient.httpGet(chapterUrl).parseHtml()
|
||||||
|
val chapterId = doc.head().getElementsByAttribute("data-chapter-id").first()!!.attr("data-chapter-id")
|
||||||
|
val json = webClient.httpGet(
|
||||||
|
"https://$domain/api/?id=$chapterId&type=chapter",
|
||||||
|
Headers.headersOf(
|
||||||
|
"Referer", chapterUrl,
|
||||||
|
"x-requested-with", "XMLHttpRequest",
|
||||||
|
),
|
||||||
|
).parseJson()
|
||||||
|
if (json.getIntOrDefault("type", 1) == 2) {
|
||||||
|
throw ParseException("Light Novels are not supported", chapterUrl)
|
||||||
|
}
|
||||||
|
val baseUrl = json.getString("baseImagesUrl")
|
||||||
|
val pages = json.getJSONArray("page_array")
|
||||||
|
return (0 until pages.length()).map { i ->
|
||||||
|
val url = concatUrl(baseUrl, pages.getString(i))
|
||||||
|
MangaPage(
|
||||||
|
id = generateUid(url),
|
||||||
|
url = url.toAbsoluteUrl(domain),
|
||||||
|
preview = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
val root = webClient.httpGet(urlBuilder().addPathSegment("manga_list").build())
|
||||||
|
.parseHtml()
|
||||||
|
.requireElementById("search_options-form")
|
||||||
|
return root.getElementsByAttributeValue("name", "categories[]")
|
||||||
|
.mapToSet { input ->
|
||||||
|
val div = input.parents().first()!!
|
||||||
|
MangaTag(
|
||||||
|
title = div.text().toTitleCase(sourceLocale),
|
||||||
|
key = input.attr("value"),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private suspend fun loadChapters(baseUrl: String, page: Int): List<MangaChapter> {
|
||||||
|
return parseChapters(webClient.httpGet("$baseUrl?limit=$page").parseHtml().body())
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun parseChapters(root: Element): List<MangaChapter> {
|
||||||
|
return root.requireElementById("chapters_content")
|
||||||
|
.select(".component-chapter").map { div ->
|
||||||
|
val a = div.selectFirstOrThrow("a")
|
||||||
|
val href = a.attrAsRelativeUrl("href")
|
||||||
|
val title = div.selectFirstOrThrow(".chapter_volume").text()
|
||||||
|
val name = div.selectFirst(".chapter_title")?.textOrNull()
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(href),
|
||||||
|
name = if (name != null && name != title) "$title: $name" else title,
|
||||||
|
number = href.substringAfterLast('/').toIntOrNull() ?: 0,
|
||||||
|
url = href,
|
||||||
|
scanlator = div.selectFirst(".team_link-name")?.textOrNull(),
|
||||||
|
uploadDate = div.selectFirst(".component-chapter-date")
|
||||||
|
?.ownTextOrNull()
|
||||||
|
.parseDate(),
|
||||||
|
branch = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun String?.parseDate(): Long {
|
||||||
|
if (this == null) {
|
||||||
|
assert(false) { "Date is null" }
|
||||||
|
return 0L
|
||||||
|
}
|
||||||
|
val parts = split(' ')
|
||||||
|
assert(parts.size == 2) { "Wrong date $this" }
|
||||||
|
val count = parts.getOrNull(0)?.toIntOrNull() ?: return 0L
|
||||||
|
val unit = parts.getOrNull(1) ?: return 0L
|
||||||
|
val calendarUnit = when (unit) {
|
||||||
|
"s" -> Calendar.SECOND
|
||||||
|
"min" -> Calendar.MINUTE
|
||||||
|
"h" -> Calendar.HOUR
|
||||||
|
"j" -> Calendar.DAY_OF_YEAR
|
||||||
|
"sem." -> Calendar.WEEK_OF_YEAR
|
||||||
|
"mois" -> Calendar.MONTH
|
||||||
|
"ans", "an" -> Calendar.YEAR
|
||||||
|
else -> {
|
||||||
|
assert(false) { "Unknown time unit $unit" }
|
||||||
|
return 0L
|
||||||
|
}
|
||||||
|
}
|
||||||
|
val calendar = Calendar.getInstance()
|
||||||
|
calendar.add(calendarUnit, -count)
|
||||||
|
return calendar.timeInMillis
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun Element.src(): String {
|
||||||
|
return attrAsAbsoluteUrlOrNull("data-cfsrc")
|
||||||
|
?: attrAsAbsoluteUrlOrNull("src")
|
||||||
|
?: attrAsAbsoluteUrlOrNull("data-src")
|
||||||
|
?: run {
|
||||||
|
assert(false) { "Image src not found" }
|
||||||
|
""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,141 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site
|
||||||
|
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.PagedMangaParser
|
||||||
|
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.text.SimpleDateFormat
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("DOUJINDESU", "DoujinDesu", "id")
|
||||||
|
class DoujinDesuParser(context: MangaLoaderContext) : PagedMangaParser(context, MangaSource.DOUJINDESU, pageSize = 18) {
|
||||||
|
|
||||||
|
override val configKeyDomain: ConfigKey.Domain
|
||||||
|
get() = ConfigKey.Domain("212.32.226.234", null)
|
||||||
|
|
||||||
|
override val sortOrders: Set<SortOrder>
|
||||||
|
get() = EnumSet.of(SortOrder.UPDATED, SortOrder.NEWEST, SortOrder.ALPHABETICAL, SortOrder.POPULARITY)
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga {
|
||||||
|
val docs = webClient.httpGet(manga.url.toAbsoluteUrl(domain)).parseHtml().selectFirstOrThrow("#archive")
|
||||||
|
val chapterDateFormat = SimpleDateFormat("EEEE, dd MMMM yyyy", sourceLocale)
|
||||||
|
val metadataEl = docs.selectFirst(".wrapper > .metadata tbody")
|
||||||
|
val state = when (metadataEl?.selectFirst("tr:contains(Status)")?.selectLast("td")?.text()) {
|
||||||
|
"Finished" -> MangaState.FINISHED
|
||||||
|
"Publishing" -> MangaState.ONGOING
|
||||||
|
else -> null
|
||||||
|
}
|
||||||
|
return manga.copy(
|
||||||
|
author = metadataEl?.selectFirst("tr:contains(Author)")?.selectLast("td")?.text(),
|
||||||
|
description = docs.selectFirst(".wrapper > .metadata > .pb-2")?.selectFirst("p")?.html(),
|
||||||
|
state = state,
|
||||||
|
rating = metadataEl?.selectFirst(".rating-prc")?.ownText()?.toFloatOrNull()?.div(10f) ?: RATING_UNKNOWN,
|
||||||
|
tags = docs.select(".tags > a").mapToSet {
|
||||||
|
MangaTag(
|
||||||
|
key = it.attr("title"),
|
||||||
|
title = it.text(),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
chapters = docs.requireElementById("chapter_list")
|
||||||
|
.select("ul > li")
|
||||||
|
.mapChapters(reversed = true) { index, element ->
|
||||||
|
val titleTag = element.selectFirstOrThrow(".epsleft > .lchx > a")
|
||||||
|
val url = titleTag.attrAsRelativeUrl("href")
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(url),
|
||||||
|
name = titleTag.text(),
|
||||||
|
number = index + 1,
|
||||||
|
url = url,
|
||||||
|
scanlator = null,
|
||||||
|
uploadDate = chapterDateFormat.tryParse(element.select(".epsleft > .date").text()),
|
||||||
|
branch = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getListPage(
|
||||||
|
page: Int,
|
||||||
|
query: String?,
|
||||||
|
tags: Set<MangaTag>?,
|
||||||
|
sortOrder: SortOrder,
|
||||||
|
): List<Manga> {
|
||||||
|
val url = urlBuilder().apply {
|
||||||
|
addPathSegment("manga")
|
||||||
|
addPathSegment("page")
|
||||||
|
addPathSegment("$page/")
|
||||||
|
val order = when (sortOrder) {
|
||||||
|
SortOrder.UPDATED -> "update"
|
||||||
|
SortOrder.POPULARITY -> "popular"
|
||||||
|
SortOrder.ALPHABETICAL -> "title"
|
||||||
|
SortOrder.NEWEST -> "latest"
|
||||||
|
else -> throw IllegalArgumentException("Sort order not supported")
|
||||||
|
}
|
||||||
|
addQueryParameter("order", order)
|
||||||
|
addQueryParameter("title", query.orEmpty())
|
||||||
|
tags?.forEach {
|
||||||
|
addEncodedQueryParameter("genre[]".urlEncoded(), it.key.urlEncoded())
|
||||||
|
}
|
||||||
|
}.build()
|
||||||
|
|
||||||
|
return webClient.httpGet(url).parseHtml()
|
||||||
|
.requireElementById("archives")
|
||||||
|
.selectFirstOrThrow("div.entries")
|
||||||
|
.select(".entry")
|
||||||
|
.map {
|
||||||
|
val titleTag = it.selectFirstOrThrow(".metadata > a")
|
||||||
|
val relativeUrl = titleTag.attrAsRelativeUrl("href")
|
||||||
|
Manga(
|
||||||
|
id = generateUid(relativeUrl),
|
||||||
|
title = titleTag.attr("title"),
|
||||||
|
altTitle = null,
|
||||||
|
url = relativeUrl,
|
||||||
|
publicUrl = relativeUrl.toAbsoluteUrl(domain),
|
||||||
|
rating = RATING_UNKNOWN,
|
||||||
|
isNsfw = true,
|
||||||
|
coverUrl = it.selectFirst(".thumbnail > img")?.attrAsAbsoluteUrl("src").orEmpty(),
|
||||||
|
tags = emptySet(),
|
||||||
|
state = null,
|
||||||
|
author = null,
|
||||||
|
largeCoverUrl = null,
|
||||||
|
description = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
val id = webClient.httpGet(chapter.url.toAbsoluteUrl(domain)).parseHtml()
|
||||||
|
.requireElementById("reader")
|
||||||
|
.attr("data-id")
|
||||||
|
return webClient.httpPost("/themes/ajax/ch.php".toAbsoluteUrl(domain), "id=$id").parseHtml()
|
||||||
|
.select("img")
|
||||||
|
.map {
|
||||||
|
val url = it.attrAsRelativeUrl("src")
|
||||||
|
MangaPage(
|
||||||
|
id = generateUid(url),
|
||||||
|
url = url,
|
||||||
|
preview = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
return webClient.httpGet("/genre/".toAbsoluteUrl(domain)).parseHtml()
|
||||||
|
.requireElementById("taxonomy")
|
||||||
|
.selectFirstOrThrow(".entries")
|
||||||
|
.select(".entry > a")
|
||||||
|
.mapToSet {
|
||||||
|
MangaTag(
|
||||||
|
key = it.attr("title"),
|
||||||
|
title = it.attr("title"),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,223 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site
|
||||||
|
|
||||||
|
import androidx.collection.ArraySet
|
||||||
|
import okhttp3.Headers
|
||||||
|
import okhttp3.Interceptor
|
||||||
|
import okhttp3.Response
|
||||||
|
import org.json.JSONArray
|
||||||
|
import org.json.JSONObject
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.PagedMangaParser
|
||||||
|
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.json.getStringOrNull
|
||||||
|
import org.koitharu.kotatsu.parsers.util.json.mapJSON
|
||||||
|
import org.koitharu.kotatsu.parsers.util.json.mapJSONIndexed
|
||||||
|
import java.text.SimpleDateFormat
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
|
||||||
|
private const val PAGE_SIZE = 20
|
||||||
|
private const val INFINITE = 999999
|
||||||
|
private const val HEADER_ENCODING = "Content-Encoding"
|
||||||
|
|
||||||
|
@MangaSourceParser("HONEYMANGA", "Honey Manga", "uk")
|
||||||
|
class HoneyMangaParser(context: MangaLoaderContext) : PagedMangaParser(context, MangaSource.HONEYMANGA, PAGE_SIZE),
|
||||||
|
Interceptor {
|
||||||
|
|
||||||
|
private val urlApi get() = "https://data.api.$domain"
|
||||||
|
private val mangaApi get() = "$urlApi/v2/manga/cursor-list"
|
||||||
|
private val chapterApi get() = "$urlApi/v2/chapter/cursor-list"
|
||||||
|
private val genresListApi get() = "$urlApi/genres-tags/genres-list"
|
||||||
|
private val framesApi get() = "$urlApi/chapter/frames"
|
||||||
|
private val searchApi get() = "https://search.api.$domain/api/v1/title/search-matching?query="
|
||||||
|
|
||||||
|
private val imageStorageUrl = "https://manga-storage.fra1.digitaloceanspaces.com/public-resources"
|
||||||
|
|
||||||
|
override val headers
|
||||||
|
get() = Headers.Builder()
|
||||||
|
.add("User-Agent", "Mozilla/5.0 (Android 13; Mobile; rv:68.0) Gecko/68.0 Firefox/109.0")
|
||||||
|
.build()
|
||||||
|
|
||||||
|
override val configKeyDomain: ConfigKey.Domain
|
||||||
|
get() = ConfigKey.Domain("honey-manga.com.ua", null)
|
||||||
|
|
||||||
|
override val sortOrders: Set<SortOrder> = EnumSet.of(
|
||||||
|
SortOrder.POPULARITY,
|
||||||
|
SortOrder.NEWEST,
|
||||||
|
)
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga {
|
||||||
|
val dateFormat = SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss", Locale.US)
|
||||||
|
val body = JSONObject()
|
||||||
|
body.put("mangaId", manga.url)
|
||||||
|
body.put("pageSize", INFINITE) // Hack lol (no)
|
||||||
|
body.put("page", 1)
|
||||||
|
body.put("sortOrder", "ASC")
|
||||||
|
val chapterRequest = webClient.httpPost(chapterApi, body).parseJson()
|
||||||
|
return manga.copy(
|
||||||
|
chapters = chapterRequest.getJSONArray("data").mapJSONIndexed() { i, jo ->
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(jo.getString("id")),
|
||||||
|
name = buildString {
|
||||||
|
append("Том ")
|
||||||
|
append(jo.optString("volume", "0"))
|
||||||
|
append(". ")
|
||||||
|
append("Розділ ")
|
||||||
|
append(jo.optString("chapterNum", "0"))
|
||||||
|
if (jo.optString("title") != "Title") {
|
||||||
|
append(" - ")
|
||||||
|
append(jo.optString("title"))
|
||||||
|
}
|
||||||
|
},
|
||||||
|
number = i + 1,
|
||||||
|
url = jo.optString("chapterResourcesId"),
|
||||||
|
scanlator = null,
|
||||||
|
uploadDate = dateFormat.tryParse(jo.getString("lastUpdated")),
|
||||||
|
branch = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getListPage(
|
||||||
|
page: Int,
|
||||||
|
query: String?,
|
||||||
|
tags: Set<MangaTag>?,
|
||||||
|
sortOrder: SortOrder,
|
||||||
|
): List<Manga> {
|
||||||
|
val body = JSONObject()
|
||||||
|
body.put("page", page)
|
||||||
|
body.put("pageSize", PAGE_SIZE)
|
||||||
|
val sort = JSONObject()
|
||||||
|
sort.put("sortBy", getSortKey(sortOrder))
|
||||||
|
sort.put("sortOrder", "DESC")
|
||||||
|
body.put("sort", sort)
|
||||||
|
|
||||||
|
val content = when {
|
||||||
|
!tags.isNullOrEmpty() -> {
|
||||||
|
// Tags
|
||||||
|
val filters = JSONArray()
|
||||||
|
val tagFilter = JSONObject()
|
||||||
|
tagFilter.put("filterBy", "genres")
|
||||||
|
tagFilter.put("filterOperator", "ALL")
|
||||||
|
val tag = JSONArray()
|
||||||
|
tags.forEach {
|
||||||
|
tag.put(it.title)
|
||||||
|
}
|
||||||
|
tagFilter.put("filterValue", tag)
|
||||||
|
filters.put(tagFilter)
|
||||||
|
body.put("filters", filters)
|
||||||
|
webClient.httpPost(mangaApi, body).parseJson().getJSONArray("data")
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
!query.isNullOrEmpty() -> {
|
||||||
|
// Search
|
||||||
|
when {
|
||||||
|
query.length < 3 -> throw IllegalArgumentException(
|
||||||
|
"The query must contain at least 3 characters (Запит має містити щонайменше 3 символи)",
|
||||||
|
)
|
||||||
|
|
||||||
|
page == searchPaginator.firstPage -> webClient
|
||||||
|
.httpGet(searchApi + query.urlEncoded())
|
||||||
|
.parseJsonArray()
|
||||||
|
|
||||||
|
else -> JSONArray()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
else -> {
|
||||||
|
// Popular/Newest
|
||||||
|
body.put("filters", JSONArray())
|
||||||
|
webClient.httpPost(mangaApi, body).parseJson().getJSONArray("data")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return content.mapJSON { jo ->
|
||||||
|
val id = jo.getString("id")
|
||||||
|
val posterUrl = jo.getString("posterUrl")
|
||||||
|
Manga(
|
||||||
|
id = generateUid(id),
|
||||||
|
title = jo.getString("title"),
|
||||||
|
altTitle = jo.getStringOrNull("alternativeTitle"),
|
||||||
|
url = id,
|
||||||
|
publicUrl = "https://$domain/book/$id",
|
||||||
|
rating = RATING_UNKNOWN,
|
||||||
|
isNsfw = isNsfw(jo.getStringOrNull("adult")),
|
||||||
|
coverUrl = getCoverUrl(posterUrl, 256),
|
||||||
|
tags = getTitleTags(jo.optJSONArray("genresAndTags")),
|
||||||
|
state = when (jo.getStringOrNull("titleStatus")) {
|
||||||
|
"Онгоінг" -> MangaState.ONGOING
|
||||||
|
"Завершено" -> MangaState.FINISHED
|
||||||
|
else -> null
|
||||||
|
},
|
||||||
|
author = null,
|
||||||
|
largeCoverUrl = getCoverUrl(posterUrl, 1080),
|
||||||
|
description = jo.getStringOrNull("description"),
|
||||||
|
chapters = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
val body = JSONObject()
|
||||||
|
body.put("chapterId", chapter.url)
|
||||||
|
val content = webClient.httpPost(framesApi, body).parseJson().getJSONObject("resourceIds")
|
||||||
|
return List(content.length()) { i ->
|
||||||
|
val item = content.getString(i.toString())
|
||||||
|
MangaPage(id = generateUid(item), "$imageStorageUrl/$item", getCoverUrl(item, 256), source)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
// https://data.api.honey-manga.com.ua/genres-tags/genres-list
|
||||||
|
val content = webClient.httpGet(genresListApi).parseJsonArray()
|
||||||
|
val tagsSet = ArraySet<MangaTag>(content.length())
|
||||||
|
repeat(content.length()) { i ->
|
||||||
|
val item = content.getString(i)
|
||||||
|
tagsSet.add(MangaTag(item, item, source))
|
||||||
|
}
|
||||||
|
return tagsSet
|
||||||
|
}
|
||||||
|
|
||||||
|
// Need for disable encoding (with encoding not working)
|
||||||
|
override fun intercept(chain: Interceptor.Chain): Response {
|
||||||
|
val request = chain.request()
|
||||||
|
val newRequest = if (request.header(HEADER_ENCODING) != null) {
|
||||||
|
request.newBuilder().removeHeader(HEADER_ENCODING).build()
|
||||||
|
} else {
|
||||||
|
request
|
||||||
|
}
|
||||||
|
return chain.proceed(newRequest)
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun isNsfw(adultValue: String?): Boolean {
|
||||||
|
val intValue = adultValue?.removeSuffix('+')?.toIntOrNull()
|
||||||
|
return intValue != null && intValue >= 18
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun getCoverUrl(id: String, w: Int): String {
|
||||||
|
// https://honey-manga.com.ua/_next/image?url=https%3A%2F%2Fhoneymangastorage.b-cdn.net%2Fpublic-resources%2F1c4613c2-ffe3-405a-b26a-2cab59ddd223%3Foptimizer%3Dimage%26width%3D512%26height%3D512&w=3840&q=75
|
||||||
|
return "https://$domain/_next/image?url=https%3A%2F%2Fhoneymangastorage.b-cdn.net%2Fpublic-resources%2F$id&w=$w&q=75"
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun getSortKey(order: SortOrder?) = when (order) {
|
||||||
|
SortOrder.POPULARITY -> "likes"
|
||||||
|
SortOrder.NEWEST -> "lastUpdated"
|
||||||
|
else -> "likes"
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun getTitleTags(jsonTags: JSONArray): Set<MangaTag> {
|
||||||
|
val tagsSet = ArraySet<MangaTag>(jsonTags.length())
|
||||||
|
repeat(jsonTags.length()) { i ->
|
||||||
|
val item = jsonTags.getString(i)
|
||||||
|
|
||||||
|
tagsSet.add(MangaTag(title = item.toTitleCase(sourceLocale), key = item, source = source))
|
||||||
|
}
|
||||||
|
return tagsSet
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,193 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site
|
||||||
|
|
||||||
|
import okhttp3.Headers
|
||||||
|
import org.json.JSONObject
|
||||||
|
import org.jsoup.nodes.Element
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.PagedMangaParser
|
||||||
|
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.ParseException
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.json.getStringOrNull
|
||||||
|
import org.koitharu.kotatsu.parsers.util.json.mapJSON
|
||||||
|
import java.text.SimpleDateFormat
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("JAPSCAN", "JapScan", "fr")
|
||||||
|
internal class JapScanParser(context: MangaLoaderContext) : PagedMangaParser(context, MangaSource.JAPSCAN, 30) {
|
||||||
|
|
||||||
|
override val sortOrders: Set<SortOrder> = EnumSet.of(SortOrder.ALPHABETICAL)
|
||||||
|
|
||||||
|
override val configKeyDomain = ConfigKey.Domain("www.japscan.lol", arrayOf("www.japscan.lol", "japscan.ws"))
|
||||||
|
|
||||||
|
override val headers: Headers = Headers.Builder()
|
||||||
|
.add("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0")
|
||||||
|
.build()
|
||||||
|
|
||||||
|
override suspend fun getListPage(
|
||||||
|
page: Int,
|
||||||
|
query: String?,
|
||||||
|
tags: Set<MangaTag>?,
|
||||||
|
sortOrder: SortOrder,
|
||||||
|
): List<Manga> {
|
||||||
|
require(tags.isNullOrEmpty()) { "Tags are not supported by this source" }
|
||||||
|
if (!query.isNullOrEmpty()) {
|
||||||
|
return if (page == paginator.firstPage) getListPageSearch(query) else emptyList()
|
||||||
|
}
|
||||||
|
val url = urlBuilder()
|
||||||
|
.addPathSegment("mangas")
|
||||||
|
.addPathSegment(page.toString())
|
||||||
|
.build()
|
||||||
|
val root = webClient.httpGet(url).parseHtml()
|
||||||
|
.requireElementById("main")
|
||||||
|
.selectFirstOrThrow(".flex-wrap")
|
||||||
|
return root.select(".mainTitle")
|
||||||
|
.map { p ->
|
||||||
|
val div = checkNotNull(p.parent())
|
||||||
|
val a = div.selectFirstOrThrow("a")
|
||||||
|
val href = a.attrAsRelativeUrl("href")
|
||||||
|
Manga(
|
||||||
|
id = generateUid(href),
|
||||||
|
title = p.text(),
|
||||||
|
altTitle = null,
|
||||||
|
url = href,
|
||||||
|
publicUrl = href.toAbsoluteUrl(domain),
|
||||||
|
rating = RATING_UNKNOWN,
|
||||||
|
isNsfw = false,
|
||||||
|
coverUrl = div.selectFirstOrThrow("img").attrAsAbsoluteUrl("src"),
|
||||||
|
tags = setOf(),
|
||||||
|
state = null,
|
||||||
|
author = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga {
|
||||||
|
val root = webClient.httpGet(manga.url.toAbsoluteUrl(domain)).parseHtml().requireElementById("main")
|
||||||
|
val dateFormat = SimpleDateFormat("dd MMM yyyy", Locale.ENGLISH)
|
||||||
|
return manga.copy(
|
||||||
|
altTitle = root.tableValue("Nom Original:"),
|
||||||
|
isNsfw = root.tableValue("Âge conseillé :")?.extractIntOrNull().let { it != null && it >= 18 },
|
||||||
|
tags = root.tableValue("Type(s):")?.split(", ")?.mapNotNullToSet {
|
||||||
|
it.toTag()
|
||||||
|
}.orEmpty() + root.tableValue("Genre(s):")?.split(", ")?.mapNotNullToSet {
|
||||||
|
it.toTag()
|
||||||
|
}.orEmpty(),
|
||||||
|
state = when (root.tableValue("Statut:")) {
|
||||||
|
"En Cours" -> MangaState.ONGOING
|
||||||
|
"Terminé", "Abondonné" -> MangaState.FINISHED
|
||||||
|
else -> null
|
||||||
|
},
|
||||||
|
author = root.tableValue("Artiste(s):")?.substringBefore(','),
|
||||||
|
description = root.selectFirst("p.list-group-item-primary")?.html(),
|
||||||
|
chapters = root.requireElementById("chapters_list")
|
||||||
|
.select("div.chapters_list")
|
||||||
|
.mapChapters(reversed = true) { i, div ->
|
||||||
|
val a = div.selectFirst("a") ?: return@mapChapters null
|
||||||
|
val href = a.attrAsRelativeUrl("href")
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(href),
|
||||||
|
name = a.text(),
|
||||||
|
number = i,
|
||||||
|
url = href,
|
||||||
|
scanlator = null,
|
||||||
|
uploadDate = dateFormat.tryParse(div.selectFirst("span.float-right")?.text()),
|
||||||
|
branch = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
val chapterUrl = chapter.url.toAbsoluteUrl(domain)
|
||||||
|
val doc = webClient.httpGet(chapterUrl).parseHtml()
|
||||||
|
val scriptUrl = doc.getElementsByTag("script").firstNotNullOf { script ->
|
||||||
|
script.attrAsAbsoluteUrlOrNull("src")?.takeIf { it.contains("/zjs/") }
|
||||||
|
}
|
||||||
|
val embeddedData = doc.requireElementById("data").attr("data-data")
|
||||||
|
val script = webClient.httpGet(scriptUrl).parseRaw()
|
||||||
|
|
||||||
|
val sample = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz".toList()
|
||||||
|
val keyRegex = Regex("""'([\dA-Z]{62})'""", RegexOption.IGNORE_CASE)
|
||||||
|
|
||||||
|
val keyTables = keyRegex.findAll(script)
|
||||||
|
.mapNotNullTo(ArrayList(2)) { match ->
|
||||||
|
match.groupValues[1].takeIf {
|
||||||
|
it.toList().sorted() == sample
|
||||||
|
}
|
||||||
|
}
|
||||||
|
check(keyTables.size == 2)
|
||||||
|
|
||||||
|
var error: Exception? = null
|
||||||
|
repeat(2) { i ->
|
||||||
|
val key = keyTables[i].zip(keyTables[1 - i]).toMap()
|
||||||
|
try {
|
||||||
|
val unscrambledData = embeddedData.map { key[it] ?: it }.joinToString("")
|
||||||
|
if (unscrambledData.startsWith("ey")) {
|
||||||
|
val array = JSONObject(context.decodeBase64(unscrambledData).toString(Charsets.UTF_8))
|
||||||
|
.getJSONArray("imagesLink")
|
||||||
|
val result = ArrayList<MangaPage>(array.length())
|
||||||
|
repeat(array.length()) { index ->
|
||||||
|
val url = array.getString(index)
|
||||||
|
result += MangaPage(
|
||||||
|
id = generateUid(url),
|
||||||
|
url = url,
|
||||||
|
preview = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
} catch (e: Exception) {
|
||||||
|
error = e
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw (error ?: ParseException("Cannot decode pages list", chapterUrl))
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
return emptySet() // not supported
|
||||||
|
}
|
||||||
|
|
||||||
|
private suspend fun getListPageSearch(
|
||||||
|
query: String,
|
||||||
|
): List<Manga> {
|
||||||
|
val json = webClient.httpPost(
|
||||||
|
"https://$domain/live-search/",
|
||||||
|
mapOf("search" to query.urlEncoded()),
|
||||||
|
).parseJsonArray()
|
||||||
|
return json.mapJSON { jo ->
|
||||||
|
val url = jo.getString("url")
|
||||||
|
Manga(
|
||||||
|
id = generateUid(url),
|
||||||
|
title = jo.getString("name"),
|
||||||
|
altTitle = jo.getStringOrNull("alternate_names")?.substringBefore(','),
|
||||||
|
url = url,
|
||||||
|
publicUrl = url.toAbsoluteUrl(domain),
|
||||||
|
rating = RATING_UNKNOWN,
|
||||||
|
isNsfw = false,
|
||||||
|
coverUrl = jo.getString("image").toAbsoluteUrl(domain),
|
||||||
|
tags = emptySet(),
|
||||||
|
state = null,
|
||||||
|
author = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun Element.tableValue(label: String): String? {
|
||||||
|
return getElementsMatchingOwnText(label).firstOrNull()?.parent()?.ownTextOrNull()
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun String.extractIntOrNull(): Int? = this.filter(Char::isDigit).toIntOrNull()
|
||||||
|
|
||||||
|
private fun String.toTag() = MangaTag(
|
||||||
|
title = this.toTitleCase(sourceLocale),
|
||||||
|
key = this.replace(' ', '-').lowercase(sourceLocale),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
@ -1,187 +0,0 @@
|
|||||||
package org.koitharu.kotatsu.parsers.site
|
|
||||||
|
|
||||||
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
|
||||||
import org.koitharu.kotatsu.parsers.MangaParser
|
|
||||||
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
|
||||||
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
|
||||||
import org.koitharu.kotatsu.parsers.model.*
|
|
||||||
import org.koitharu.kotatsu.parsers.util.*
|
|
||||||
import java.text.SimpleDateFormat
|
|
||||||
import java.util.*
|
|
||||||
|
|
||||||
@Deprecated("Permanently shutdown")
|
|
||||||
@MangaSourceParser("MANGAOWL", "MangaOwl", "en")
|
|
||||||
internal class MangaOwlParser(override val context: MangaLoaderContext) : MangaParser(MangaSource.MANGAOWL) {
|
|
||||||
|
|
||||||
override val configKeyDomain = ConfigKey.Domain("mangaowls.com", null)
|
|
||||||
|
|
||||||
override val sortOrders: Set<SortOrder> = EnumSet.of(
|
|
||||||
SortOrder.POPULARITY,
|
|
||||||
SortOrder.NEWEST,
|
|
||||||
SortOrder.UPDATED,
|
|
||||||
)
|
|
||||||
|
|
||||||
private val regexNsfw = Regex("(yaoi)|(yuri)|(smut)|(mature)|(adult)", RegexOption.IGNORE_CASE)
|
|
||||||
|
|
||||||
override suspend fun getList(
|
|
||||||
offset: Int,
|
|
||||||
query: String?,
|
|
||||||
tags: Set<MangaTag>?,
|
|
||||||
sortOrder: SortOrder,
|
|
||||||
): List<Manga> {
|
|
||||||
val page = (offset / 36f).toIntUp().inc()
|
|
||||||
val link = buildString {
|
|
||||||
append("https://")
|
|
||||||
append(getDomain())
|
|
||||||
when {
|
|
||||||
!query.isNullOrEmpty() -> {
|
|
||||||
append("/search/$page?search=")
|
|
||||||
append(query.urlEncoded())
|
|
||||||
}
|
|
||||||
|
|
||||||
!tags.isNullOrEmpty() -> {
|
|
||||||
for (tag in tags) {
|
|
||||||
append(tag.key)
|
|
||||||
}
|
|
||||||
append("/$page?type=${getAlternativeSortKey(sortOrder)}")
|
|
||||||
}
|
|
||||||
|
|
||||||
else -> {
|
|
||||||
append("/${getSortKey(sortOrder)}/$page")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
val doc = context.httpGet(link).parseHtml()
|
|
||||||
val slides = doc.body().selectOrThrow("ul.slides")
|
|
||||||
val items = slides.select("div.col-md-2")
|
|
||||||
return items.mapNotNull { item ->
|
|
||||||
val href = item.selectFirst("h6 a")?.attrAsRelativeUrlOrNull("href") ?: return@mapNotNull null
|
|
||||||
Manga(
|
|
||||||
id = generateUid(href),
|
|
||||||
title = item.selectFirst("h6 a")?.text() ?: return@mapNotNull null,
|
|
||||||
coverUrl = item.select("div.img-responsive").attr("abs:data-background-image"),
|
|
||||||
altTitle = null,
|
|
||||||
author = null,
|
|
||||||
rating = runCatching {
|
|
||||||
item.selectFirst("div.block-stars")
|
|
||||||
?.text()
|
|
||||||
?.toFloatOrNull()
|
|
||||||
?.div(10f)
|
|
||||||
}.getOrNull() ?: RATING_UNKNOWN,
|
|
||||||
url = href,
|
|
||||||
isNsfw = false,
|
|
||||||
tags = emptySet(),
|
|
||||||
state = null,
|
|
||||||
publicUrl = href.toAbsoluteUrl(getDomain()),
|
|
||||||
source = source,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
override suspend fun getDetails(manga: Manga): Manga {
|
|
||||||
val doc = context.httpGet(manga.publicUrl).parseHtml()
|
|
||||||
val info = doc.body().selectFirstOrThrow("div.single_detail")
|
|
||||||
val table = doc.body().selectFirstOrThrow("div.single-grid-right")
|
|
||||||
val dateFormat = SimpleDateFormat("MM/dd/yyyy", Locale.US)
|
|
||||||
val trRegex = "window\\['tr'] = '([^']*)';".toRegex(RegexOption.IGNORE_CASE)
|
|
||||||
val trElement = doc.getElementsByTag("script").find { trRegex.find(it.data()) != null }
|
|
||||||
?: doc.parseFailed("Oops, tr not found")
|
|
||||||
val tr = trRegex.find(trElement.data())!!.groups[1]!!.value
|
|
||||||
val s = context.encodeBase64(getDomain().toByteArray())
|
|
||||||
var isNsfw = manga.isNsfw
|
|
||||||
val parsedTags = info.select("div.col-xs-12.col-md-8.single-right-grid-right > p > a[href*=genres]")
|
|
||||||
.mapNotNullToSet {
|
|
||||||
val a = it.selectFirst("a") ?: return@mapNotNullToSet null
|
|
||||||
val name = a.text()
|
|
||||||
if (!isNsfw && isNsfwGenre(name)) {
|
|
||||||
isNsfw = true
|
|
||||||
}
|
|
||||||
MangaTag(
|
|
||||||
title = name.toTitleCase(),
|
|
||||||
key = a.attr("href"),
|
|
||||||
source = source,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return manga.copy(
|
|
||||||
description = info.selectFirst(".description")?.html(),
|
|
||||||
largeCoverUrl = info.select("img").first()?.let { img ->
|
|
||||||
if (img.hasAttr("data-src")) img.attr("abs:data-src") else img.attr("abs:src")
|
|
||||||
},
|
|
||||||
isNsfw = isNsfw,
|
|
||||||
author = info.selectFirst("p.fexi_header_para a.author_link")?.text(),
|
|
||||||
state = parseStatus(info.select("p.fexi_header_para:contains(status)").first()?.ownText()),
|
|
||||||
tags = manga.tags + parsedTags,
|
|
||||||
chapters = table.select("div.table.table-chapter-list").select("li.list-group-item.chapter_list")
|
|
||||||
.asReversed().mapChapters { i, li ->
|
|
||||||
val a = li.select("a")
|
|
||||||
val href = a.attr("data-href").ifEmpty {
|
|
||||||
li.parseFailed("Link is missing")
|
|
||||||
}
|
|
||||||
MangaChapter(
|
|
||||||
id = generateUid(href),
|
|
||||||
name = a.select("label").text(),
|
|
||||||
number = i + 1,
|
|
||||||
url = "$href?tr=$tr&s=$s",
|
|
||||||
scanlator = null,
|
|
||||||
branch = null,
|
|
||||||
uploadDate = dateFormat.tryParse(li.selectFirst("small:last-of-type")?.text()),
|
|
||||||
source = MangaSource.MANGAOWL,
|
|
||||||
)
|
|
||||||
},
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
|
||||||
val fullUrl = chapter.url.toAbsoluteUrl(getDomain())
|
|
||||||
val doc = context.httpGet(fullUrl).parseHtml()
|
|
||||||
val root = doc.body().selectOrThrow("div.item img.owl-lazy")
|
|
||||||
return root.map { div ->
|
|
||||||
val url = div?.attrAsRelativeUrlOrNull("data-src") ?: doc.parseFailed("Page image not found")
|
|
||||||
MangaPage(
|
|
||||||
id = generateUid(url),
|
|
||||||
url = url,
|
|
||||||
preview = null,
|
|
||||||
referer = url,
|
|
||||||
source = MangaSource.MANGAOWL,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private fun parseStatus(status: String?) = when {
|
|
||||||
status == null -> null
|
|
||||||
status.contains("Ongoing") -> MangaState.ONGOING
|
|
||||||
status.contains("Completed") -> MangaState.FINISHED
|
|
||||||
else -> null
|
|
||||||
}
|
|
||||||
|
|
||||||
override suspend fun getTags(): Set<MangaTag> {
|
|
||||||
val doc = context.httpGet("https://${getDomain()}/").parseHtml()
|
|
||||||
val root = doc.body().select("ul.dropdown-menu.multi-column.columns-3").select("li")
|
|
||||||
return root.mapToSet { p ->
|
|
||||||
val a = p.selectFirstOrThrow("a")
|
|
||||||
MangaTag(
|
|
||||||
title = a.text().toTitleCase(),
|
|
||||||
key = a.attr("href"),
|
|
||||||
source = source,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private fun getSortKey(sortOrder: SortOrder) =
|
|
||||||
when (sortOrder) {
|
|
||||||
SortOrder.POPULARITY -> "popular"
|
|
||||||
SortOrder.NEWEST -> "new_release"
|
|
||||||
SortOrder.UPDATED -> "lastest"
|
|
||||||
else -> "lastest"
|
|
||||||
}
|
|
||||||
|
|
||||||
private fun getAlternativeSortKey(sortOrder: SortOrder) =
|
|
||||||
when (sortOrder) {
|
|
||||||
SortOrder.POPULARITY -> "0"
|
|
||||||
SortOrder.NEWEST -> "2"
|
|
||||||
SortOrder.UPDATED -> "3"
|
|
||||||
else -> "3"
|
|
||||||
}
|
|
||||||
|
|
||||||
private fun isNsfwGenre(name: String): Boolean = regexNsfw.containsMatchIn(name)
|
|
||||||
}
|
|
||||||
@ -0,0 +1,196 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site
|
||||||
|
|
||||||
|
import androidx.collection.ArrayMap
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.PagedMangaParser
|
||||||
|
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("MANHWA18", "Manhwa18", "en")
|
||||||
|
class Manhwa18Parser(context: MangaLoaderContext) :
|
||||||
|
PagedMangaParser(context, MangaSource.MANHWA18, pageSize = 20, searchPageSize = 20) {
|
||||||
|
|
||||||
|
override val configKeyDomain: ConfigKey.Domain
|
||||||
|
get() = ConfigKey.Domain("manhwa18.net", null)
|
||||||
|
|
||||||
|
override val sortOrders: Set<SortOrder>
|
||||||
|
get() = EnumSet.of(SortOrder.UPDATED, SortOrder.POPULARITY, SortOrder.ALPHABETICAL, SortOrder.NEWEST)
|
||||||
|
|
||||||
|
private val tagsMap = SuspendLazy(::parseTags)
|
||||||
|
|
||||||
|
override suspend fun getFavicons(): Favicons {
|
||||||
|
return Favicons(
|
||||||
|
listOf(
|
||||||
|
Favicon("https://${domain}/uploads/logos/logo-mini.png", 92, null),
|
||||||
|
),
|
||||||
|
domain,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga {
|
||||||
|
val docs = webClient.httpGet(manga.url.toAbsoluteUrl(domain)).parseHtml()
|
||||||
|
val cardInfoElement = docs.selectFirst(".card .manga-info")
|
||||||
|
val author = cardInfoElement?.selectFirst("b:contains(Author(s))")?.parent()
|
||||||
|
?.select("a.btn")
|
||||||
|
?.joinToString(", ") { it.text() }
|
||||||
|
val availableTags = tagsMap.get()
|
||||||
|
val tags = cardInfoElement?.selectFirst("b:contains(Genre(s))")?.parent()
|
||||||
|
?.select("a.btn")
|
||||||
|
?.mapNotNullToSet { availableTags[it.text().lowercase(Locale.ENGLISH)] }
|
||||||
|
val state = cardInfoElement?.selectFirst("b:contains(Status)")?.parent()
|
||||||
|
?.selectFirst("a.btn")
|
||||||
|
?.let {
|
||||||
|
when (it.text()) {
|
||||||
|
"On going" -> MangaState.ONGOING
|
||||||
|
"Completed" -> MangaState.FINISHED
|
||||||
|
else -> null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return manga.copy(
|
||||||
|
altTitle = cardInfoElement?.selectFirst("b:contains(Other names)")?.parent()?.ownText()?.removePrefix(": "),
|
||||||
|
author = author,
|
||||||
|
description = docs.selectFirst(".series-summary .summary-content")?.html(),
|
||||||
|
tags = tags.orEmpty(),
|
||||||
|
state = state,
|
||||||
|
chapters = docs.select(".card-body > .list-chapters > a").mapChapters(reversed = true) { index, element ->
|
||||||
|
// attrAsRelativeUrl only return page url without the '/'
|
||||||
|
val chapterUrl = element.attrAsAbsoluteUrlOrNull("href")?.toRelativeUrl(domain)
|
||||||
|
?: return@mapChapters null
|
||||||
|
val uploadDate = parseUploadDate(element.selectFirst(".chapter-time")?.text())
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(chapterUrl),
|
||||||
|
name = element.selectFirst(".chapter-name")?.text().orEmpty(),
|
||||||
|
number = index + 1,
|
||||||
|
url = chapterUrl,
|
||||||
|
scanlator = null,
|
||||||
|
uploadDate = uploadDate,
|
||||||
|
branch = null,
|
||||||
|
source = MangaSource.MANHWA18,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 7 minutes ago
|
||||||
|
// 5 hours ago
|
||||||
|
// 2 days ago
|
||||||
|
// 2 weeks ago
|
||||||
|
// 4 years ago
|
||||||
|
private fun parseUploadDate(timeStr: String?): Long {
|
||||||
|
timeStr ?: return 0
|
||||||
|
|
||||||
|
val timeWords = timeStr.split(' ')
|
||||||
|
if (timeWords.size != 3) return 0
|
||||||
|
val timeWord = timeWords[1]
|
||||||
|
val timeAmount = timeWords[0].toIntOrNull() ?: return 0
|
||||||
|
val timeUnit = when (timeWord) {
|
||||||
|
"minute", "minutes" -> Calendar.MINUTE
|
||||||
|
"hour", "hours" -> Calendar.HOUR
|
||||||
|
"day", "days" -> Calendar.DAY_OF_YEAR
|
||||||
|
"week", "weeks" -> Calendar.WEEK_OF_YEAR
|
||||||
|
"month", "months" -> Calendar.MONTH
|
||||||
|
"year", "years" -> Calendar.YEAR
|
||||||
|
else -> return 0
|
||||||
|
}
|
||||||
|
val cal = Calendar.getInstance()
|
||||||
|
cal.add(timeUnit, -timeAmount)
|
||||||
|
return cal.time.time
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getListPage(
|
||||||
|
page: Int,
|
||||||
|
query: String?,
|
||||||
|
tags: Set<MangaTag>?,
|
||||||
|
sortOrder: SortOrder,
|
||||||
|
): List<Manga> {
|
||||||
|
val sortQuery = when (sortOrder) {
|
||||||
|
SortOrder.ALPHABETICAL -> "az"
|
||||||
|
SortOrder.POPULARITY -> "top"
|
||||||
|
SortOrder.UPDATED -> "update"
|
||||||
|
SortOrder.NEWEST -> "new"
|
||||||
|
else -> ""
|
||||||
|
}
|
||||||
|
|
||||||
|
val tagQuery = tags?.joinToString(",") { it.key }.orEmpty()
|
||||||
|
val url = buildString {
|
||||||
|
append("https://")
|
||||||
|
append(domain)
|
||||||
|
append("/tim-kiem?page=")
|
||||||
|
append(page)
|
||||||
|
if (!query.isNullOrEmpty()) {
|
||||||
|
append("&q=")
|
||||||
|
append(query.urlEncoded())
|
||||||
|
}
|
||||||
|
append("&accept_genres=$tagQuery")
|
||||||
|
append("&sort=")
|
||||||
|
append(sortQuery)
|
||||||
|
}
|
||||||
|
|
||||||
|
val docs = webClient.httpGet(url).parseHtml()
|
||||||
|
|
||||||
|
return docs.select(".card-body .thumb-item-flow")
|
||||||
|
.map {
|
||||||
|
val titleElement = it.selectFirstOrThrow(".thumb_attr.series-title > a")
|
||||||
|
val absUrl = titleElement.attrAsAbsoluteUrl("href")
|
||||||
|
Manga(
|
||||||
|
id = generateUid(absUrl.toRelativeUrl(domain)),
|
||||||
|
title = titleElement.text(),
|
||||||
|
altTitle = null,
|
||||||
|
url = absUrl.toRelativeUrl(domain),
|
||||||
|
publicUrl = absUrl,
|
||||||
|
rating = RATING_UNKNOWN,
|
||||||
|
isNsfw = true,
|
||||||
|
coverUrl = it.selectFirst("div.img-in-ratio")?.attrAsAbsoluteUrl("data-bg").orEmpty(),
|
||||||
|
tags = emptySet(),
|
||||||
|
state = null,
|
||||||
|
author = null,
|
||||||
|
largeCoverUrl = null,
|
||||||
|
description = null,
|
||||||
|
source = MangaSource.MANHWA18,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
val chapterUrl = chapter.url.toAbsoluteUrl(domain)
|
||||||
|
val doc = webClient.httpGet(chapterUrl).parseHtml()
|
||||||
|
return doc.requireElementById("chapter-content").select("img").mapNotNull {
|
||||||
|
val url = it.attrAsRelativeUrlOrNull("data-src")
|
||||||
|
?: it.attrAsRelativeUrlOrNull("src")
|
||||||
|
?: return@mapNotNull null
|
||||||
|
MangaPage(
|
||||||
|
id = generateUid(url),
|
||||||
|
url = url,
|
||||||
|
preview = null,
|
||||||
|
source = MangaSource.MANHWA18,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
return tagsMap.get().values.toSet()
|
||||||
|
}
|
||||||
|
|
||||||
|
private suspend fun parseTags(): Map<String, MangaTag> {
|
||||||
|
val doc = webClient.httpGet("https://$domain/tim-kiem?q=").parseHtml()
|
||||||
|
val list = doc.getElementsByAttribute("data-genre-id")
|
||||||
|
if (list.isEmpty()) {
|
||||||
|
return emptyMap()
|
||||||
|
}
|
||||||
|
val result = ArrayMap<String, MangaTag>(list.size)
|
||||||
|
for (item in list) {
|
||||||
|
val id = item.attr("data-genre-id")
|
||||||
|
val name = item.text()
|
||||||
|
result[name.lowercase(Locale.ENGLISH)] = MangaTag(
|
||||||
|
title = name.toTitleCase(Locale.ENGLISH),
|
||||||
|
key = id,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,153 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site
|
||||||
|
|
||||||
|
import androidx.collection.ArrayMap
|
||||||
|
import androidx.collection.ArraySet
|
||||||
|
import kotlinx.coroutines.sync.Mutex
|
||||||
|
import kotlinx.coroutines.sync.withLock
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.PagedMangaParser
|
||||||
|
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.text.SimpleDateFormat
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("TRUYENTRANHLH", "TruyentranhLH", "vi")
|
||||||
|
class TruyentranhLHParser(context: MangaLoaderContext) :
|
||||||
|
PagedMangaParser(context, source = MangaSource.TRUYENTRANHLH, pageSize = 18) {
|
||||||
|
|
||||||
|
override val configKeyDomain: ConfigKey.Domain = ConfigKey.Domain("truyentranhlh.net", null)
|
||||||
|
override val sortOrders: Set<SortOrder> = EnumSet.allOf(SortOrder::class.java)
|
||||||
|
|
||||||
|
private val mutex = Mutex()
|
||||||
|
private var tagCache: Map<String, MangaTag>? = null
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga {
|
||||||
|
val docs = webClient.httpGet(manga.url.toAbsoluteUrl(domain)).parseHtml()
|
||||||
|
val infoHeaderEl = docs.selectFirst("main.section-body")
|
||||||
|
val infoEl = docs.selectFirst("main.section-body .series-information")
|
||||||
|
val tags = infoEl?.select(".info-item:contains(Thể loại) > .info-value > a")?.mapNotNullToSet {
|
||||||
|
getOrCreateTagMap()[it.text().trim()]
|
||||||
|
}
|
||||||
|
val state = when (infoEl?.selectFirst(".info-item:contains(Tình trạng) > .info-value")?.text()) {
|
||||||
|
"Đang tiến hành" -> MangaState.ONGOING
|
||||||
|
"Đã hoàn thành" -> MangaState.FINISHED
|
||||||
|
else -> null
|
||||||
|
}
|
||||||
|
val rating = infoHeaderEl?.let {
|
||||||
|
val like = it.selectFirst("#like .block.feature-name")?.text()?.toIntOrNull()
|
||||||
|
val disLike = it.selectFirst("#dislike .block.feature-name")?.text()?.toIntOrNull()
|
||||||
|
when {
|
||||||
|
like == null || disLike == null -> RATING_UNKNOWN
|
||||||
|
like == 0 && disLike == 0 -> RATING_UNKNOWN
|
||||||
|
else -> like.toFloat() / (like + disLike)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
val chapterDateFormat = SimpleDateFormat("dd/MM/yyyy", Locale.US)
|
||||||
|
|
||||||
|
return manga.copy(
|
||||||
|
altTitle = infoEl?.selectFirst(".info-item:contains(Tên khác) > .info-value")?.text(),
|
||||||
|
author = infoEl?.select(".info-item:contains(Tác giả) > .info-value")?.joinToString { it.text() },
|
||||||
|
tags = tags ?: emptySet(),
|
||||||
|
state = state,
|
||||||
|
rating = rating ?: RATING_UNKNOWN,
|
||||||
|
description = infoHeaderEl?.selectFirst(".series-summary .summary-content")?.html(),
|
||||||
|
chapters = docs.select("ul.list-chapters.at-series > a").mapChapters(reversed = true) { index, element ->
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(element.attrAsRelativeUrl("href")),
|
||||||
|
name = element.selectFirst(".chapter-name")?.text()?.trim().orEmpty(),
|
||||||
|
number = index + 1,
|
||||||
|
url = element.attrAsRelativeUrl("href"),
|
||||||
|
scanlator = null,
|
||||||
|
uploadDate = chapterDateFormat.tryParse(element.selectFirst(".chapter-time")?.text()),
|
||||||
|
branch = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getListPage(
|
||||||
|
page: Int,
|
||||||
|
query: String?,
|
||||||
|
tags: Set<MangaTag>?,
|
||||||
|
sortOrder: SortOrder,
|
||||||
|
): List<Manga> {
|
||||||
|
val sortQuery = when (sortOrder) {
|
||||||
|
SortOrder.UPDATED -> "update"
|
||||||
|
SortOrder.NEWEST -> "new"
|
||||||
|
SortOrder.RATING -> "like"
|
||||||
|
SortOrder.POPULARITY -> "top"
|
||||||
|
SortOrder.ALPHABETICAL -> "az"
|
||||||
|
}
|
||||||
|
val url = urlBuilder().apply {
|
||||||
|
addPathSegment("tim-kiem")
|
||||||
|
addQueryParameter("sort", sortQuery)
|
||||||
|
addQueryParameter("page", page.toString())
|
||||||
|
if (!query.isNullOrEmpty()) {
|
||||||
|
addQueryParameter("q", query)
|
||||||
|
}
|
||||||
|
if (!tags.isNullOrEmpty()) {
|
||||||
|
val tagsQuery = tags.joinToString(separator = ",") { it.key }
|
||||||
|
addEncodedQueryParameter("accept_genres", tagsQuery)
|
||||||
|
}
|
||||||
|
}.build()
|
||||||
|
|
||||||
|
return webClient.httpGet(url).parseHtml()
|
||||||
|
.select(".container .card.card-dark .row > .thumb-item-flow")
|
||||||
|
.mapNotNull {
|
||||||
|
val a = it.selectFirstOrThrow(".thumb-wrapper a")
|
||||||
|
Manga(
|
||||||
|
id = generateUid(a.attrAsRelativeUrl("href")),
|
||||||
|
url = a.attrAsRelativeUrl("href"),
|
||||||
|
publicUrl = a.attrAsAbsoluteUrl("href"),
|
||||||
|
title = it.select(".thumb_attr.series-title").text(),
|
||||||
|
altTitle = null,
|
||||||
|
rating = RATING_UNKNOWN,
|
||||||
|
isNsfw = false,
|
||||||
|
coverUrl = a.selectFirst("div[data-bg]")?.attrAsAbsoluteUrl("data-bg").orEmpty(),
|
||||||
|
tags = emptySet(),
|
||||||
|
state = null,
|
||||||
|
author = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
val url = chapter.url.toAbsoluteUrl(domain)
|
||||||
|
return webClient.httpGet(url).parseHtml().select("#chapter-content > img").mapNotNull {
|
||||||
|
val imageUrl = it.attrAsRelativeUrlOrNull("data-src")
|
||||||
|
?: it.attrAsRelativeUrlOrNull("src")
|
||||||
|
?: return@mapNotNull null
|
||||||
|
MangaPage(
|
||||||
|
id = generateUid(imageUrl),
|
||||||
|
url = imageUrl,
|
||||||
|
preview = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
return ArraySet(getOrCreateTagMap().values)
|
||||||
|
}
|
||||||
|
|
||||||
|
private suspend fun getOrCreateTagMap(): Map<String, MangaTag> = mutex.withLock {
|
||||||
|
tagCache?.let { return it }
|
||||||
|
val docs = webClient.httpGet("/tim-kiem".toAbsoluteUrl(domain)).parseHtml()
|
||||||
|
val tags = docs.select(".search-border-left .row > .search-gerne_item").mapNotNull {
|
||||||
|
MangaTag(
|
||||||
|
title = it.text().trim(),
|
||||||
|
key = it.selectFirst("label[data-genre-id]")
|
||||||
|
?.attr("data-genre-id")
|
||||||
|
?.trim() ?: return@mapNotNull null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
val tagMap = tags.associateByTo(ArrayMap(tags.size)) { it.title }
|
||||||
|
tagCache = tagMap
|
||||||
|
return tagMap
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,53 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site.grouple
|
||||||
|
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.config.ConfigKey
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.AuthRequiredException
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.ParseException
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaChapter
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaPage
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaSource
|
||||||
|
import org.koitharu.kotatsu.parsers.util.domain
|
||||||
|
import org.koitharu.kotatsu.parsers.util.parseFailed
|
||||||
|
import org.koitharu.kotatsu.parsers.util.parseHtml
|
||||||
|
import org.koitharu.kotatsu.parsers.util.urlEncoded
|
||||||
|
|
||||||
|
@MangaSourceParser("ALLHENTAI", "ALlHentai", "ru")
|
||||||
|
internal class AllHentaiParser(
|
||||||
|
context: MangaLoaderContext,
|
||||||
|
) : GroupleParser(context, MangaSource.ALLHENTAI, 1) {
|
||||||
|
|
||||||
|
override val configKeyDomain = ConfigKey.Domain(
|
||||||
|
"2023.allhen.online",
|
||||||
|
null,
|
||||||
|
)
|
||||||
|
override val defaultIsNsfw = true
|
||||||
|
|
||||||
|
override val authUrl: String
|
||||||
|
get() {
|
||||||
|
val targetUri = "https://${domain}/".urlEncoded()
|
||||||
|
return "https://qawa.org/internal/auth/login?targetUri=$targetUri&siteId=1"
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
try {
|
||||||
|
return super.getPages(chapter)
|
||||||
|
} catch (e: ParseException) {
|
||||||
|
if (isAuthorized) {
|
||||||
|
throw e
|
||||||
|
} else {
|
||||||
|
throw AuthRequiredException(source)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getUsername(): String {
|
||||||
|
val root = webClient.httpGet("https://qawa.org/").parseHtml().body()
|
||||||
|
val element = root.selectFirst("img.user-avatar") ?: throw AuthRequiredException(source)
|
||||||
|
val res = element.parent()?.text()
|
||||||
|
return if (res.isNullOrEmpty()) {
|
||||||
|
root.parseFailed("Cannot find username")
|
||||||
|
} else res
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,69 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site.madara
|
||||||
|
|
||||||
|
import org.jsoup.nodes.Document
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.model.Manga
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaChapter
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaSource
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaTag
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.text.SimpleDateFormat
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("HENTAI_4FREE", "Hentai4Free", "en")
|
||||||
|
internal class Hentai4Free(context: MangaLoaderContext) :
|
||||||
|
MadaraParser(context, MangaSource.HENTAI_4FREE, "hentai4free.net") {
|
||||||
|
|
||||||
|
override val tagPrefix = "hentai-tag/"
|
||||||
|
|
||||||
|
override val isNsfwSource = true
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
val doc = webClient.httpGet("https://$domain/").parseHtml()
|
||||||
|
val body = doc.body()
|
||||||
|
val root1 = body.selectFirst("header")?.selectFirst("ul.second-menu")
|
||||||
|
val list = root1?.select("li").orEmpty()
|
||||||
|
val keySet = HashSet<String>(list.size)
|
||||||
|
return list.mapNotNullToSet { li ->
|
||||||
|
val a = li.selectFirst("a") ?: return@mapNotNullToSet null
|
||||||
|
val href = a.attr("href").removeSuffix("/")
|
||||||
|
.substringAfterLast(tagPrefix, "")
|
||||||
|
if (href.isEmpty() || !keySet.add(href)) {
|
||||||
|
return@mapNotNullToSet null
|
||||||
|
}
|
||||||
|
MangaTag(
|
||||||
|
key = href,
|
||||||
|
title = a.ownText().trim().toTitleCase(),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getChapters(manga: Manga, doc: Document): List<MangaChapter> {
|
||||||
|
val slug = manga.url.removeSuffix('/').substringAfterLast('/')
|
||||||
|
val doc2 = webClient.httpPost(
|
||||||
|
"https://$domain/hentai/$slug/ajax/chapters/",
|
||||||
|
mapOf(),
|
||||||
|
).parseHtml()
|
||||||
|
val ul = doc2.body().selectFirstOrThrow("ul")
|
||||||
|
val dateFormat = SimpleDateFormat(datePattern, Locale.US)
|
||||||
|
return ul.select("li").mapChapters(reversed = true) { i, li ->
|
||||||
|
val a = li.selectFirst("a")
|
||||||
|
val href = a?.attrAsRelativeUrlOrNull("href") ?: li.parseFailed("Link is missing")
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(href),
|
||||||
|
name = a.ownText(),
|
||||||
|
number = i + 1,
|
||||||
|
url = href,
|
||||||
|
uploadDate = parseChapterDate(
|
||||||
|
dateFormat,
|
||||||
|
li.selectFirst("span.chapter-release-date i")?.text(),
|
||||||
|
),
|
||||||
|
source = source,
|
||||||
|
scanlator = null,
|
||||||
|
branch = null,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,81 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site.madara
|
||||||
|
|
||||||
|
import org.jsoup.nodes.Document
|
||||||
|
import org.jsoup.nodes.Element
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.model.Manga
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaChapter
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaSource
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaTag
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.text.SimpleDateFormat
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("ISEKAISCAN_EU", "IsekaiScan", "en")
|
||||||
|
internal class IsekaiScanEuParser(context: MangaLoaderContext) :
|
||||||
|
MadaraParser(context, MangaSource.ISEKAISCAN_EU, "isekaiscan.to") {
|
||||||
|
|
||||||
|
override val datePattern = "MM/dd/yyyy"
|
||||||
|
|
||||||
|
override suspend fun getChapters(manga: Manga, doc: Document): List<MangaChapter> {
|
||||||
|
doc.selectFirst("ul.version-chap")?.let {
|
||||||
|
return parseChapters(it)
|
||||||
|
}
|
||||||
|
val mangaId = doc.body().requireElementById("manga-chapters-holder").attr("data-id")
|
||||||
|
val ul = webClient.httpPost(
|
||||||
|
"https://${domain}/wp-admin/admin-ajax.php",
|
||||||
|
mapOf(
|
||||||
|
"action" to "manga_get_chapters",
|
||||||
|
"manga" to mangaId,
|
||||||
|
),
|
||||||
|
).parseHtml().body().selectFirstOrThrow("ul")
|
||||||
|
return parseChapters(ul)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
val doc = webClient.httpGet("https://$domain/mangax/").parseHtml()
|
||||||
|
val body = doc.body()
|
||||||
|
val root1 = body.selectFirst("header")?.selectFirst("ul.second-menu")
|
||||||
|
val root2 = body.selectFirst("div.genres_wrap")?.selectFirst("ul.list-unstyled")
|
||||||
|
if (root1 == null && root2 == null) {
|
||||||
|
doc.parseFailed("Root not found")
|
||||||
|
}
|
||||||
|
val list = root1?.select("li").orEmpty() + root2?.select("li").orEmpty()
|
||||||
|
val keySet = HashSet<String>(list.size)
|
||||||
|
return list.mapNotNullToSet { li ->
|
||||||
|
val a = li.selectFirst("a") ?: return@mapNotNullToSet null
|
||||||
|
val href = a.attr("href").removeSuffix("/")
|
||||||
|
.substringAfterLast(tagPrefix, "")
|
||||||
|
if (href.isEmpty() || !keySet.add(href)) {
|
||||||
|
return@mapNotNullToSet null
|
||||||
|
}
|
||||||
|
MangaTag(
|
||||||
|
key = href,
|
||||||
|
title = a.ownText().toTitleCase(Locale.ENGLISH),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun parseChapters(ul: Element): List<MangaChapter> {
|
||||||
|
val dateFormat = SimpleDateFormat(datePattern, Locale.US)
|
||||||
|
return ul.select("li").mapChapters(reversed = true) { i, li ->
|
||||||
|
val a = li.selectFirst("a")
|
||||||
|
val href = a?.attrAsRelativeUrlOrNull("href") ?: li.parseFailed("Link is missing")
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(href),
|
||||||
|
name = a.ownText(),
|
||||||
|
number = i + 1,
|
||||||
|
url = href,
|
||||||
|
uploadDate = parseChapterDate(
|
||||||
|
dateFormat,
|
||||||
|
li.selectFirst("span.chapter-release-date i")?.text(),
|
||||||
|
),
|
||||||
|
source = source,
|
||||||
|
scanlator = null,
|
||||||
|
branch = null,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,120 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site.madara
|
||||||
|
|
||||||
|
import androidx.collection.arraySetOf
|
||||||
|
import org.jsoup.nodes.Element
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("MANGAREAD", "MangaRead", "en")
|
||||||
|
internal class MangaRead(context: MangaLoaderContext) :
|
||||||
|
MadaraParser(context, MangaSource.MANGAREAD, "www.mangaread.org") {
|
||||||
|
|
||||||
|
override val tagPrefix = "genres/"
|
||||||
|
override val datePattern = "dd.MM.yyyy"
|
||||||
|
private val nsfwTags = arraySetOf("yaoi", "yuri", "mature")
|
||||||
|
|
||||||
|
init {
|
||||||
|
paginator.firstPage = 1
|
||||||
|
searchPaginator.firstPage = 1
|
||||||
|
}
|
||||||
|
|
||||||
|
override val sortOrders: Set<SortOrder> = EnumSet.of(
|
||||||
|
SortOrder.UPDATED,
|
||||||
|
SortOrder.POPULARITY,
|
||||||
|
SortOrder.ALPHABETICAL,
|
||||||
|
SortOrder.RATING,
|
||||||
|
SortOrder.POPULARITY,
|
||||||
|
)
|
||||||
|
|
||||||
|
override suspend fun getListPage(
|
||||||
|
page: Int,
|
||||||
|
query: String?,
|
||||||
|
tags: Set<MangaTag>?,
|
||||||
|
sortOrder: SortOrder,
|
||||||
|
): List<Manga> {
|
||||||
|
val url = urlBuilder()
|
||||||
|
.addPathSegment("page")
|
||||||
|
.addPathSegment(page.toString())
|
||||||
|
.addQueryParameter("s", query.orEmpty())
|
||||||
|
.addQueryParameter("post_type", "wp-manga")
|
||||||
|
tags?.forEachIndexed { index, tag ->
|
||||||
|
url.addQueryParameter("genre[$index]", tag.key)
|
||||||
|
}
|
||||||
|
url.addQueryParameter("op", "")
|
||||||
|
.addQueryParameter("author", "")
|
||||||
|
.addQueryParameter("artist", "")
|
||||||
|
.addQueryParameter("release", "")
|
||||||
|
.addQueryParameter("adult", "")
|
||||||
|
if (query.isNullOrEmpty()) {
|
||||||
|
url.addQueryParameter(
|
||||||
|
"&m_orderby=",
|
||||||
|
when (sortOrder) {
|
||||||
|
SortOrder.RATING -> "trending"
|
||||||
|
SortOrder.ALPHABETICAL -> "alphabet"
|
||||||
|
SortOrder.POPULARITY -> "views"
|
||||||
|
SortOrder.NEWEST -> "new-manga"
|
||||||
|
SortOrder.UPDATED -> "latest"
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
val root = webClient.httpGet(url.build()).parseHtml().body().selectFirstOrThrow(".search-wrap")
|
||||||
|
return root.select(".c-tabs-item__content").map { div ->
|
||||||
|
val a = div.selectFirstOrThrow("a")
|
||||||
|
val img = div.selectLastOrThrow("img")
|
||||||
|
val href = a.attrAsRelativeUrl("href")
|
||||||
|
val postContent = root.selectFirstOrThrow(".post-content")
|
||||||
|
val tagSet = postContent.getElementsContainingOwnText("Genre")
|
||||||
|
.firstOrNull()?.tableValue()
|
||||||
|
?.getElementsByAttributeValueContaining("href", tagPrefix)
|
||||||
|
?.mapToSet { it.asMangaTag() }.orEmpty()
|
||||||
|
Manga(
|
||||||
|
id = generateUid(href),
|
||||||
|
title = a.attr("title"),
|
||||||
|
altTitle = postContent.getElementsContainingOwnText("Alternative")
|
||||||
|
.firstOrNull()?.tableValue()?.text()?.trim(),
|
||||||
|
url = href,
|
||||||
|
publicUrl = a.attrAsAbsoluteUrl("href"),
|
||||||
|
coverUrl = img.src().orEmpty(),
|
||||||
|
author = postContent.getElementsContainingOwnText("Author")
|
||||||
|
.firstOrNull()?.tableValue()?.text()?.trim(),
|
||||||
|
state = postContent.getElementsContainingOwnText("Status")
|
||||||
|
.firstOrNull()?.tableValue()?.text()?.asMangaState(),
|
||||||
|
isNsfw = isNsfw(tagSet),
|
||||||
|
rating = div.selectFirstOrThrow(".score").text()
|
||||||
|
.toFloatOrNull()?.div(5f) ?: RATING_UNKNOWN,
|
||||||
|
tags = tagSet,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun Element.tableValue(): Element {
|
||||||
|
for (p in parents()) {
|
||||||
|
val children = p.children()
|
||||||
|
if (children.size == 2) {
|
||||||
|
return children[1]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
parseFailed("Cannot find tableValue for node ${text()}")
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun isNsfw(tags: Set<MangaTag>): Boolean {
|
||||||
|
return tags.any { it.key in nsfwTags }
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun String.asMangaState() = when (trim().lowercase(sourceLocale)) {
|
||||||
|
"ongoing" -> MangaState.ONGOING
|
||||||
|
"completed" -> MangaState.FINISHED
|
||||||
|
else -> null
|
||||||
|
}
|
||||||
|
|
||||||
|
private fun Element.asMangaTag() = MangaTag(
|
||||||
|
title = ownText(),
|
||||||
|
key = attr("href").removeSuffix('/').substringAfterLast('/')
|
||||||
|
.replace('-', '+'),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
@ -0,0 +1,46 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site.madara
|
||||||
|
|
||||||
|
import kotlinx.coroutines.async
|
||||||
|
import kotlinx.coroutines.coroutineScope
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.ParseException
|
||||||
|
import org.koitharu.kotatsu.parsers.model.Manga
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaSource
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaTag
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("MANGALINK_AR", "Mangalink", "ar")
|
||||||
|
internal class MangalinkParser(context: MangaLoaderContext) :
|
||||||
|
MadaraParser(context, MangaSource.MANGALINK_AR, "mangalink.online") {
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga = coroutineScope {
|
||||||
|
val fullUrl = manga.url.toAbsoluteUrl(domain)
|
||||||
|
val doc = webClient.httpGet(fullUrl).parseHtml()
|
||||||
|
val chaptersDeferred = async { getChapters(manga, doc) }
|
||||||
|
val root = doc.body().selectFirst("div.profile-manga")
|
||||||
|
?.selectFirst("div.summary_content")
|
||||||
|
?.selectFirst("div.post-content")
|
||||||
|
?: throw ParseException("Root not found", fullUrl)
|
||||||
|
val root2 = doc.body().selectFirst("div.content-area")
|
||||||
|
?.selectFirst("div.c-page")
|
||||||
|
?: throw ParseException("Root2 not found", fullUrl)
|
||||||
|
manga.copy(
|
||||||
|
tags = root.selectFirst("div.genres-content")?.select("a")
|
||||||
|
?.mapNotNullToSet { a ->
|
||||||
|
MangaTag(
|
||||||
|
key = a.attr("href").removeSuffix("/").substringAfterLast('/'),
|
||||||
|
title = a.text().toTitleCase(),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
} ?: manga.tags,
|
||||||
|
description = root2.selectFirst("div.description-summary")
|
||||||
|
?.selectFirst("div.summary__content")
|
||||||
|
?.select("p")
|
||||||
|
?.filterNot { it.ownText().startsWith("A brief description") }
|
||||||
|
?.joinToString { it.html() },
|
||||||
|
chapters = chaptersDeferred.await(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
@ -1,56 +0,0 @@
|
|||||||
package org.koitharu.kotatsu.parsers.site.madara
|
|
||||||
|
|
||||||
import org.jsoup.nodes.Element
|
|
||||||
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
|
||||||
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
|
||||||
import org.koitharu.kotatsu.parsers.model.Manga
|
|
||||||
import org.koitharu.kotatsu.parsers.model.MangaChapter
|
|
||||||
import org.koitharu.kotatsu.parsers.model.MangaSource
|
|
||||||
import org.koitharu.kotatsu.parsers.model.MangaState
|
|
||||||
import org.koitharu.kotatsu.parsers.util.attrAsAbsoluteUrlOrNull
|
|
||||||
import org.koitharu.kotatsu.parsers.util.mapToSet
|
|
||||||
import org.koitharu.kotatsu.parsers.util.selectFirstOrThrow
|
|
||||||
import java.util.*
|
|
||||||
|
|
||||||
@MangaSourceParser("MANGAS_ORIGINES", "Mangas Origines", "fr")
|
|
||||||
internal class MangasOriginesParser(context: MangaLoaderContext) :
|
|
||||||
Madara6Parser(context, MangaSource.MANGAS_ORIGINES, "mangas-origines.fr") {
|
|
||||||
|
|
||||||
override val tagPrefix = "catalogues-genre/"
|
|
||||||
|
|
||||||
override fun getFaviconUrl(): String {
|
|
||||||
return "https://${getDomain()}/wp-content/uploads/2020/11/Mangas-150x150.png"
|
|
||||||
}
|
|
||||||
|
|
||||||
override fun parseDetails(manga: Manga, body: Element, chapters: List<MangaChapter>): Manga {
|
|
||||||
val root = body.selectFirstOrThrow(".site-content")
|
|
||||||
val postContent = root.selectFirstOrThrow(".post-content")
|
|
||||||
val tags = postContent.getElementsContainingOwnText("Genre")
|
|
||||||
.firstOrNull()?.tableValue()
|
|
||||||
?.getElementsByAttributeValueContaining("href", tagPrefix)
|
|
||||||
?.mapToSet { a -> a.asMangaTag() } ?: manga.tags
|
|
||||||
return manga.copy(
|
|
||||||
largeCoverUrl = root.selectFirst("picture")
|
|
||||||
?.selectFirst("img[data-src]")
|
|
||||||
?.attrAsAbsoluteUrlOrNull("data-src"),
|
|
||||||
description = (root.selectFirst(".detail-content")
|
|
||||||
?: root.selectFirstOrThrow(".manga-excerpt")).html(),
|
|
||||||
author = postContent.getElementsContainingOwnText("Auteur")
|
|
||||||
.firstOrNull()?.tableValue()?.text()?.trim(),
|
|
||||||
state = postContent.getElementsContainingOwnText("STATUS")
|
|
||||||
.firstOrNull()?.tableValue()?.text()?.asMangaState(),
|
|
||||||
tags = tags,
|
|
||||||
isNsfw = body.hasClass("adult-content"),
|
|
||||||
chapters = chapters,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
override fun String.asMangaState() = when (trim().lowercase(Locale.FRANCE)) {
|
|
||||||
"en cours" -> MangaState.ONGOING
|
|
||||||
"abandonné",
|
|
||||||
"terminé",
|
|
||||||
-> MangaState.FINISHED
|
|
||||||
|
|
||||||
else -> null
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@ -0,0 +1,95 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site.madara
|
||||||
|
|
||||||
|
import kotlinx.coroutines.async
|
||||||
|
import kotlinx.coroutines.coroutineScope
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.ParseException
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.text.SimpleDateFormat
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("NEATMANGA", "NeatManga", "en")
|
||||||
|
internal class NeatManga(context: MangaLoaderContext) : MadaraParser(context, MangaSource.NEATMANGA, "neatmangas.com") {
|
||||||
|
|
||||||
|
override val datePattern = "dd MMMM yyyy"
|
||||||
|
|
||||||
|
override suspend fun getDetails(manga: Manga): Manga = coroutineScope {
|
||||||
|
val chaptersDeferred = async { getChapters(manga) }
|
||||||
|
val fullUrl = manga.url.toAbsoluteUrl(domain)
|
||||||
|
val doc = webClient.httpGet(fullUrl).parseHtml()
|
||||||
|
val root = doc.body().selectFirst("div.profile-manga")
|
||||||
|
?.selectFirst("div.summary_content")
|
||||||
|
?.selectFirst("div.post-content")
|
||||||
|
?: throw ParseException("Root not found", fullUrl)
|
||||||
|
val root2 = doc.body().selectFirst("div.content-area")
|
||||||
|
?.selectFirst("div.c-page")
|
||||||
|
?: throw ParseException("Root2 not found", fullUrl)
|
||||||
|
manga.copy(
|
||||||
|
tags = root.selectFirst("div.genres-content")?.select("a")
|
||||||
|
?.mapNotNullToSet { a ->
|
||||||
|
MangaTag(
|
||||||
|
key = a.attr("href").removeSuffix("/").substringAfterLast('/'),
|
||||||
|
title = a.text().toTitleCase(),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
} ?: manga.tags,
|
||||||
|
description = root2.getElementsMatchingOwnText("Summary")
|
||||||
|
.firstOrNull()
|
||||||
|
?.nextElementSibling()
|
||||||
|
?.select("p")
|
||||||
|
?.filterNot { it.ownText().startsWith("A brief description") }
|
||||||
|
?.joinToString { it.html() },
|
||||||
|
chapters = chaptersDeferred.await(),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private suspend fun getChapters(manga: Manga): List<MangaChapter> {
|
||||||
|
val slug = manga.url.removeSuffix('/').substringAfterLast('/')
|
||||||
|
val doc2 = webClient.httpPost(
|
||||||
|
"https://$domain/manga/$slug/ajax/chapters/",
|
||||||
|
mapOf(),
|
||||||
|
).parseHtml()
|
||||||
|
val ul = doc2.body().selectFirstOrThrow("ul")
|
||||||
|
val dateFormat = SimpleDateFormat(datePattern, Locale.US)
|
||||||
|
return ul.select("li").mapChapters(reversed = true) { i, li ->
|
||||||
|
val a = li.selectFirst("a")
|
||||||
|
val href = a?.attrAsRelativeUrlOrNull("href") ?: li.parseFailed("Link is missing")
|
||||||
|
MangaChapter(
|
||||||
|
id = generateUid(href),
|
||||||
|
name = a.ownText(),
|
||||||
|
number = i + 1,
|
||||||
|
url = href,
|
||||||
|
uploadDate = parseChapterDate(
|
||||||
|
dateFormat,
|
||||||
|
li.selectFirst("span.chapter-release-date i")?.text(),
|
||||||
|
),
|
||||||
|
source = source,
|
||||||
|
scanlator = null,
|
||||||
|
branch = null,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getPages(chapter: MangaChapter): List<MangaPage> {
|
||||||
|
val fullUrl = chapter.url.toAbsoluteUrl(domain)
|
||||||
|
val doc = webClient.httpGet(fullUrl).parseHtml()
|
||||||
|
val root = doc.body().selectFirst("div.main-col-inner")
|
||||||
|
?.selectFirst("div.reading-content")
|
||||||
|
?: throw ParseException("Root not found", fullUrl)
|
||||||
|
return root.select("div.page-break").mapNotNull { div ->
|
||||||
|
val img = div.selectFirst("img")
|
||||||
|
if (img == null || img.attr("id").isNullOrEmpty()) {
|
||||||
|
return@mapNotNull null
|
||||||
|
}
|
||||||
|
val url = img.src()?.toRelativeUrl(domain) ?: div.parseFailed("Image src not found")
|
||||||
|
MangaPage(
|
||||||
|
id = generateUid(url),
|
||||||
|
url = url,
|
||||||
|
preview = null,
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,86 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.site.madara
|
||||||
|
|
||||||
|
import org.jsoup.nodes.Element
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaLoaderContext
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaSourceParser
|
||||||
|
import org.koitharu.kotatsu.parsers.model.*
|
||||||
|
import org.koitharu.kotatsu.parsers.util.*
|
||||||
|
import java.util.*
|
||||||
|
|
||||||
|
@MangaSourceParser("REAPER_SCANS_ID", "ReaperScansID", "in")
|
||||||
|
internal class ReaperScansParser(context: MangaLoaderContext) :
|
||||||
|
Madara6Parser(context, MangaSource.REAPER_SCANS_ID, "reaperscans.id") {
|
||||||
|
|
||||||
|
override val datePattern = "MMMM dd, yyyy"
|
||||||
|
override val tagPrefix = "genre/"
|
||||||
|
override val sourceLocale: Locale = Locale.ENGLISH
|
||||||
|
|
||||||
|
override fun String.asMangaState(): MangaState? = when (this) {
|
||||||
|
"OnGoing",
|
||||||
|
"Upcoming",
|
||||||
|
-> MangaState.ONGOING
|
||||||
|
|
||||||
|
"Completed",
|
||||||
|
"Dropped",
|
||||||
|
-> MangaState.FINISHED
|
||||||
|
|
||||||
|
else -> null
|
||||||
|
}
|
||||||
|
|
||||||
|
override fun parseDetails(manga: Manga, body: Element, chapters: List<MangaChapter>): Manga {
|
||||||
|
val root = body.selectFirstOrThrow(".site-content")
|
||||||
|
val postContent = root.requireElementById("nav-info")
|
||||||
|
val tags = postContent.getElementsContainingOwnText("Gênero")
|
||||||
|
.firstOrNull()?.tableValue()
|
||||||
|
?.getElementsByAttributeValueContaining("href", tagPrefix)
|
||||||
|
?.mapToSet { a -> a.asMangaTag() } ?: manga.tags
|
||||||
|
return manga.copy(
|
||||||
|
rating = postContent.selectFirstOrThrow(".post-rating")
|
||||||
|
.selectFirstOrThrow(".total_votes").text().toFloat() / 5f,
|
||||||
|
largeCoverUrl = root.selectFirst(".summary_image")
|
||||||
|
?.selectFirst("img[data-src]")
|
||||||
|
?.attrAsAbsoluteUrlOrNull("data-src")
|
||||||
|
.assertNotNull("largeCoverUrl"),
|
||||||
|
description = root.requireElementById("nav-profile")
|
||||||
|
.selectFirstOrThrow(".description-summary")
|
||||||
|
.firstElementChild()?.html(),
|
||||||
|
author = postContent.getElementsContainingOwnText("Author(s)")
|
||||||
|
.firstOrNull()?.tableValue()?.text()?.trim(),
|
||||||
|
altTitle = postContent.getElementsContainingOwnText("Alternative")
|
||||||
|
.firstOrNull()?.tableValue()?.text()?.trim(),
|
||||||
|
state = postContent.getElementsContainingOwnText("Status")
|
||||||
|
.firstOrNull()?.tableValue()?.text()?.asMangaState(),
|
||||||
|
tags = tags,
|
||||||
|
isNsfw = body.hasClass("adult-content"),
|
||||||
|
chapters = chapters,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
override suspend fun getTags(): Set<MangaTag> {
|
||||||
|
val doc = webClient.httpGet("https://${domain}/semua-komik/").parseHtml()
|
||||||
|
val body = doc.body()
|
||||||
|
val root1 = body.selectFirst("header")?.selectFirst("ul.second-menu")
|
||||||
|
val root2 = body.selectFirst("div.genres_wrap")?.selectFirst("ul.list-unstyled")
|
||||||
|
if (root1 == null && root2 == null) {
|
||||||
|
doc.parseFailed("Root not found")
|
||||||
|
}
|
||||||
|
val list = root1?.select("li").orEmpty() + root2?.select("li").orEmpty()
|
||||||
|
val keySet = HashSet<String>(list.size)
|
||||||
|
return list.mapNotNullToSet { li ->
|
||||||
|
val a = li.selectFirst("a") ?: return@mapNotNullToSet null
|
||||||
|
val href = a.attr("href").removeSuffix("/")
|
||||||
|
.substringAfterLast(tagPrefix, "")
|
||||||
|
if (href.isEmpty() || !keySet.add(href)) {
|
||||||
|
return@mapNotNullToSet null
|
||||||
|
}
|
||||||
|
MangaTag(
|
||||||
|
key = href,
|
||||||
|
title = a.ownText().trim().ifEmpty {
|
||||||
|
a.selectFirst(".menu-image-title")?.text()?.trim() ?: return@mapNotNullToSet null
|
||||||
|
}.toTitleCase(),
|
||||||
|
source = source,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
@ -0,0 +1,35 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.util
|
||||||
|
|
||||||
|
import kotlinx.coroutines.Deferred
|
||||||
|
import kotlinx.coroutines.Job
|
||||||
|
import kotlinx.coroutines.flow.channelFlow
|
||||||
|
import kotlinx.coroutines.flow.first
|
||||||
|
import kotlinx.coroutines.launch
|
||||||
|
import kotlin.coroutines.cancellation.CancellationException
|
||||||
|
|
||||||
|
fun Iterable<Job>.cancelAll(cause: CancellationException? = null) {
|
||||||
|
forEach { it.cancel(cause) }
|
||||||
|
}
|
||||||
|
|
||||||
|
suspend fun <T> Iterable<Deferred<T>>.awaitFirst(): T {
|
||||||
|
return channelFlow {
|
||||||
|
for (deferred in this@awaitFirst) {
|
||||||
|
launch {
|
||||||
|
send(deferred.await())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}.first().also { this@awaitFirst.cancelAll() }
|
||||||
|
}
|
||||||
|
|
||||||
|
suspend fun <T> Collection<Deferred<T>>.awaitFirst(condition: (T) -> Boolean): T {
|
||||||
|
return channelFlow {
|
||||||
|
for (deferred in this@awaitFirst) {
|
||||||
|
launch {
|
||||||
|
val result = deferred.await()
|
||||||
|
if (condition(result)) {
|
||||||
|
send(result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}.first().also { this@awaitFirst.cancelAll() }
|
||||||
|
}
|
||||||
@ -0,0 +1,78 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.util
|
||||||
|
|
||||||
|
import okhttp3.HttpUrl
|
||||||
|
import org.jsoup.nodes.Element
|
||||||
|
import org.koitharu.kotatsu.parsers.InternalParsersApi
|
||||||
|
import org.koitharu.kotatsu.parsers.MangaParser
|
||||||
|
import org.koitharu.kotatsu.parsers.exception.ParseException
|
||||||
|
import org.koitharu.kotatsu.parsers.model.Manga
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaChapter
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaPage
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaTag
|
||||||
|
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create a unique id for [Manga]/[MangaChapter]/[MangaPage].
|
||||||
|
* @param url must be relative url, without a domain
|
||||||
|
* @see [Manga.id]
|
||||||
|
* @see [MangaChapter.id]
|
||||||
|
* @see [MangaPage.id]
|
||||||
|
*/
|
||||||
|
@InternalParsersApi
|
||||||
|
fun MangaParser.generateUid(url: String): Long {
|
||||||
|
var h = 1125899906842597L
|
||||||
|
source.name.forEach { c ->
|
||||||
|
h = 31 * h + c.code
|
||||||
|
}
|
||||||
|
url.forEach { c ->
|
||||||
|
h = 31 * h + c.code
|
||||||
|
}
|
||||||
|
return h
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Create a unique id for [Manga]/[MangaChapter]/[MangaPage].
|
||||||
|
* @param id an internal identifier
|
||||||
|
* @see [Manga.id]
|
||||||
|
* @see [MangaChapter.id]
|
||||||
|
* @see [MangaPage.id]
|
||||||
|
*/
|
||||||
|
@InternalParsersApi
|
||||||
|
fun MangaParser.generateUid(id: Long): Long {
|
||||||
|
var h = 1125899906842597L
|
||||||
|
source.name.forEach { c ->
|
||||||
|
h = 31 * h + c.code
|
||||||
|
}
|
||||||
|
h = 31 * h + id
|
||||||
|
return h
|
||||||
|
}
|
||||||
|
|
||||||
|
@InternalParsersApi
|
||||||
|
fun Element.parseFailed(message: String? = null): Nothing {
|
||||||
|
throw ParseException(message, ownerDocument()?.location() ?: baseUri(), null)
|
||||||
|
}
|
||||||
|
|
||||||
|
@InternalParsersApi
|
||||||
|
fun Set<MangaTag>?.oneOrThrowIfMany(): MangaTag? {
|
||||||
|
return when {
|
||||||
|
isNullOrEmpty() -> null
|
||||||
|
size == 1 -> first()
|
||||||
|
else -> throw IllegalArgumentException("Multiple genres are not supported by this source")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
val MangaParser.domain: String
|
||||||
|
get() {
|
||||||
|
return config[configKeyDomain]
|
||||||
|
}
|
||||||
|
|
||||||
|
fun MangaParser.getDomain(subdomain: String): String {
|
||||||
|
val domain = domain
|
||||||
|
return subdomain + "." + domain.removePrefix("www.")
|
||||||
|
}
|
||||||
|
|
||||||
|
fun MangaParser.urlBuilder(): HttpUrl.Builder {
|
||||||
|
return HttpUrl.Builder()
|
||||||
|
.scheme("https")
|
||||||
|
.host(domain)
|
||||||
|
}
|
||||||
@ -0,0 +1,13 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.util
|
||||||
|
|
||||||
|
import kotlinx.coroutines.CancellationException
|
||||||
|
|
||||||
|
inline fun <T, R> T.runCatchingCancellable(block: T.() -> R): Result<R> {
|
||||||
|
return try {
|
||||||
|
Result.success(block())
|
||||||
|
} catch (e: CancellationException) {
|
||||||
|
throw e
|
||||||
|
} catch (e: Throwable) {
|
||||||
|
Result.failure(e)
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,36 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers.util
|
||||||
|
|
||||||
|
import kotlinx.coroutines.sync.Mutex
|
||||||
|
import kotlinx.coroutines.sync.withLock
|
||||||
|
|
||||||
|
class SuspendLazy<T>(
|
||||||
|
private val initializer: suspend () -> T,
|
||||||
|
) {
|
||||||
|
|
||||||
|
private val mutex = Mutex()
|
||||||
|
private var cachedValue: Any? = Uninitialized
|
||||||
|
|
||||||
|
@Suppress("UNCHECKED_CAST")
|
||||||
|
suspend fun get(): T {
|
||||||
|
// fast way
|
||||||
|
cachedValue.let {
|
||||||
|
if (it !== Uninitialized) {
|
||||||
|
return it as T
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return mutex.withLock {
|
||||||
|
cachedValue.let {
|
||||||
|
if (it !== Uninitialized) {
|
||||||
|
return it as T
|
||||||
|
}
|
||||||
|
}
|
||||||
|
val result = initializer()
|
||||||
|
cachedValue = result
|
||||||
|
result
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
suspend fun tryGet() = runCatchingCancellable { get() }
|
||||||
|
|
||||||
|
private object Uninitialized
|
||||||
|
}
|
||||||
@ -0,0 +1,49 @@
|
|||||||
|
package org.koitharu.kotatsu.parsers
|
||||||
|
|
||||||
|
import okhttp3.Interceptor
|
||||||
|
import okhttp3.Request
|
||||||
|
import okhttp3.Response
|
||||||
|
import org.koitharu.kotatsu.parsers.model.MangaSource
|
||||||
|
import org.koitharu.kotatsu.parsers.util.domain
|
||||||
|
import org.koitharu.kotatsu.parsers.util.mergeWith
|
||||||
|
import java.net.IDN
|
||||||
|
|
||||||
|
private const val HEADER_USER_AGENT = "User-Agent"
|
||||||
|
private const val HEADER_REFERER = "Referer"
|
||||||
|
|
||||||
|
internal class CommonHeadersInterceptor(
|
||||||
|
private val userAgent: String,
|
||||||
|
) : Interceptor {
|
||||||
|
|
||||||
|
override fun intercept(chain: Interceptor.Chain): Response {
|
||||||
|
val request = chain.request()
|
||||||
|
val source = request.tag(MangaSource::class.java)
|
||||||
|
val parser = source?.newParser(MangaLoaderContextMock)
|
||||||
|
val sourceHeaders = parser?.headers
|
||||||
|
val headersBuilder = request.headers.newBuilder()
|
||||||
|
if (sourceHeaders != null) {
|
||||||
|
headersBuilder.mergeWith(sourceHeaders, replaceExisting = false)
|
||||||
|
}
|
||||||
|
if (headersBuilder[HEADER_USER_AGENT] == null) {
|
||||||
|
headersBuilder[HEADER_USER_AGENT] = userAgent
|
||||||
|
}
|
||||||
|
if (headersBuilder[HEADER_REFERER] == null && parser != null) {
|
||||||
|
val idn = IDN.toASCII(parser.domain)
|
||||||
|
headersBuilder[HEADER_REFERER] = "https://$idn/"
|
||||||
|
}
|
||||||
|
val newRequest = request.newBuilder().headers(headersBuilder.build()).build()
|
||||||
|
return if (parser is Interceptor) {
|
||||||
|
parser.intercept(ProxyChain(chain, newRequest))
|
||||||
|
} else {
|
||||||
|
return chain.proceed(newRequest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private class ProxyChain(
|
||||||
|
private val delegate: Interceptor.Chain,
|
||||||
|
private val request: Request,
|
||||||
|
) : Interceptor.Chain by delegate {
|
||||||
|
|
||||||
|
override fun request(): Request = request
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,20 +0,0 @@
|
|||||||
package org.koitharu.kotatsu.parsers
|
|
||||||
|
|
||||||
import okhttp3.Interceptor
|
|
||||||
import okhttp3.Response
|
|
||||||
|
|
||||||
private const val HEADER_USER_AGENT = "User-Agent"
|
|
||||||
internal class UserAgentInterceptor(
|
|
||||||
private val userAgent: String,
|
|
||||||
) : Interceptor {
|
|
||||||
|
|
||||||
override fun intercept(chain: Interceptor.Chain): Response {
|
|
||||||
val request = chain.request()
|
|
||||||
val newRequest = if (request.header(HEADER_USER_AGENT) == null) {
|
|
||||||
request.newBuilder().header(HEADER_USER_AGENT, userAgent).build()
|
|
||||||
} else {
|
|
||||||
request
|
|
||||||
}
|
|
||||||
return chain.proceed(newRequest)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Loading…
Reference in New Issue