scrapper-api

Cronologia

Nikola Ignjatovic d724149647 Init commit		4 anni fa
..
source	Init commit	4 anni fa
license	Init commit	4 anni fa
package.json	Init commit	4 anni fa
readme.md	Init commit	4 anni fa

Huge thanks to for sponsoring me!

Simplified HTTP requests

Got is a human-friendly and powerful HTTP request library.

It was created because the popular request package is bloated:

Got is for Node.js. For browsers, we recommend Ky.

Highlights

Promise & stream API
Request cancelation
RFC compliant caching
Follows redirects
Retries on failure
Progress events
Handles gzip/deflate
Timeout handling
Errors with metadata
JSON mode
WHATWG URL support
Hooks
Instances with custom defaults
Composable
Electron support
Used by ~2000 packages and ~500K repos
Actively maintained

Moving from Request?

See how Got compares to other HTTP libraries

Install

$ npm install got

<img src="https://c5.patreon.com/external/logo/become_a_patron_button@2x.png" width="160">

Usage

const got = require('got');

(async () => {
	try {
		const response = await got('sindresorhus.com');
		console.log(response.body);
		//=> '<!doctype html> ...'
	} catch (error) {
		console.log(error.response.body);
		//=> 'Internal server error ...'
	}
})();

Streams

const fs = require('fs');
const got = require('got');

got.stream('sindresorhus.com').pipe(fs.createWriteStream('index.html'));

// For POST, PUT, and PATCH methods `got.stream` returns a `stream.Writable`
fs.createReadStream('index.html').pipe(got.stream.post('sindresorhus.com'));

API

It’s a GET request by default, but can be changed by using different methods or in the options.

got(url, [options])

Returns a Promise for a response object or a stream if options.stream is set to true.

url

Type: string Object

The URL to request, as a string, a https.request options object, or a WHATWG URL.

Properties from options will override properties in the parsed url.

If no protocol is specified, it will default to https.

options

Type: Object

Any of the https.request options.

baseUrl

Type: string Object

When specified, url will be prepended by baseUrl.
If you specify an absolute URL, it will skip the baseUrl.

Very useful when used with got.extend() to create niche-specific Got instances.

Can be a string or a WHATWG URL.

Slash at the end of baseUrl and at the beginning of the url argument is optional:

await got('hello', {baseUrl: 'https://example.com/v1'});
//=> 'https://example.com/v1/hello'

await got('/hello', {baseUrl: 'https://example.com/v1/'});
//=> 'https://example.com/v1/hello'

await got('/hello', {baseUrl: 'https://example.com/v1'});
//=> 'https://example.com/v1/hello'

headers

Type: Object
Default: {}

Request headers.

Existing headers will be overwritten. Headers set to null will be omitted.

stream

Type: boolean
Default: false

Returns a Stream instead of a Promise. This is equivalent to calling got.stream(url, [options]).

body

Type: string Buffer stream.Readable form-data instance

Note: If you provide this option, got.stream() will be read-only.

The body that will be sent with a POST request.

If present in options and options.method is not set, options.method will be set to POST.

The content-length header will be automatically set if body is a string / Buffer / fs.createReadStream instance / form-data instance, and content-length and transfer-encoding are not manually set in options.headers.

cookieJar

Type: tough.CookieJar instance

Note: If you provide this option, options.headers.cookie will be overridden.

Cookie support. You don’t have to care about parsing or how to store them. Example.

encoding

Type: string null
Default: 'utf8'

Encoding to be used on setEncoding of the response data. If null, the body is returned as a Buffer (binary data).

form

Type: boolean
Default: false

Note: If you provide this option, got.stream() will be read-only. Note: body must be a plain object. It will be converted to a query string using (new URLSearchParams(object)).toString().

If set to true and Content-Type header is not set, it will be set to application/x-www-form-urlencoded.

json

Type: boolean
Default: false

Note: If you use got.stream(), this option will be ignored. Note: body must be a plain object or array and will be stringified.

If set to true and Content-Type header is not set, it will be set to application/json.

Parse response body with JSON.parse and set accept header to application/json. If used in conjunction with the form option, the body will the stringified as querystring and the response parsed as JSON.

query

Type: string Object<string, string|number> URLSearchParams

Query string that will be added to the request URL. This will override the query string in url.

If you need to pass in an array, you can do it using a URLSearchParams instance:

const got = require('got');

const query = new URLSearchParams([['key', 'a'], ['key', 'b']]);

got('https://example.com', {query});

console.log(query.toString());
//=> 'key=a&key=b'

And if you need a different array format, you could use the query-string package:

const got = require('got');
const queryString = require('query-string');

const query = queryString.stringify({key: ['a', 'b']}, {arrayFormat: 'bracket'});

got('https://example.com', {query});

console.log(query);
//=> 'key[]=a&key[]=b'

timeout

Type: number Object

Milliseconds to wait for the server to end the response before aborting the request with got.TimeoutError error (a.k.a. request property). By default, there’s no timeout.

This also accepts an object with the following fields to constrain the duration of each phase of the request lifecycle:

lookup starts when a socket is assigned and ends when the hostname has been resolved. Does not apply when using a Unix domain socket.
connect starts when lookup completes (or when the socket is assigned if lookup does not apply to the request) and ends when the socket is connected.
secureConnect starts when connect completes and ends when the handshaking process completes (HTTPS only).
socket starts when the socket is connected. See request.setTimeout.
response starts when the request has been written to the socket and ends when the response headers are received.
send starts when the socket is connected and ends with the request has been written to the socket.
request starts when the request is initiated and ends when the response’s end event fires.

retry

Type: number Object
Default:

retries: 2
methods: GET PUT HEAD DELETE OPTIONS TRACE
statusCodes: 408 413 429 500 502 503 504
maxRetryAfter: undefined
errorCodes: ETIMEDOUT ECONNRESET EADDRINUSE ECONNREFUSED EPIPE ENOTFOUND ENETUNREACH EAI_AGAIN

An object representing retries, methods, statusCodes, maxRetryAfter and errorCodes fields for the time until retry, allowed methods, allowed status codes, maximum Retry-After time and allowed error codes.

If maxRetryAfter is set to undefined, it will use options.timeout.
If Retry-After header is greater than maxRetryAfter, it will cancel the request.

Delays between retries counts with function 1000 * Math.pow(2, retry) + Math.random() * 100, where retry is attempt number (starts from 1).

The retries property can be a number or a function with retry and error arguments. The function must return a delay in milliseconds (0 return value cancels retry).

By default, it retries only on the specified methods, status codes, and on these network errors:

ETIMEDOUT: One of the timeout limits were reached.
ECONNRESET: Connection was forcibly closed by a peer.
EADDRINUSE: Could not bind to any free port.
ECONNREFUSED: Connection was refused by the server.
EPIPE: The remote side of the stream being written has been closed.
ENOTFOUND: Couldn’t resolve the hostname to an IP address.
ENETUNREACH: No internet connection.
EAI_AGAIN: DNS lookup timed out.

followRedirect

Type: boolean
Default: true

Defines if redirect responses should be followed automatically.

Note that if a 303 is sent by the server in response to any request type (POST, DELETE, etc.), Got will automatically request the resource pointed to in the location header via GET. This is in accordance with the spec.

decompress

Type: boolean
Default: true

Decompress the response automatically. This will set the accept-encoding header to gzip, deflate unless you set it yourself.

If this is disabled, a compressed response is returned as a Buffer. This may be useful if you want to handle decompression yourself or stream the raw compressed data.

cache

Type: Object
Default: false

Cache adapter instance for storing cached data.

request

Type: Function
Default: http.request https.request (depending on the protocol)

Custom request function. The main purpose of this is to support HTTP2 using a wrapper.

useElectronNet

Type: boolean
Default: false

When used in Electron, Got will use electron.net instead of the Node.js http module. According to the Electron docs, it should be fully compatible, but it’s not entirely. See #443 and #461.

throwHttpErrors

Type: boolean
Default: true

Determines if a got.HTTPError is thrown for error responses (non-2xx status codes).

If this is disabled, requests that encounter an error status code will be resolved with the response instead of throwing. This may be useful if you are checking for resource availability and are expecting error responses.

agent

Same as the agent option for http.request, but with an extra feature:

If you require different agents for different protocols, you can pass a map of agents to the agent option. This is necessary because a request to one protocol might redirect to another. In such a scenario, Got will switch over to the right protocol agent for you.

const got = require('got');
const HttpAgent = require('agentkeepalive');
const {HttpsAgent} = HttpAgent;

got('sindresorhus.com', {
	agent: {
		http: new HttpAgent(),
		https: new HttpsAgent()
	}
});

hooks

Type: Object<string, Function[]>

Hooks allow modifications during the request lifecycle. Hook functions may be async and are run serially.