外貿(mào)網(wǎng)站建設(shè)，讓業(yè)務(wù)全球可達(dá)
0%
其實(shí)有一個很重要的原因就是現(xiàn)在的爬蟲太多，現(xiàn)在很多AI的爬蟲也全網(wǎng)爬行。
特別是國外的爬蟲。
一查詢高頻率訪問的IP地址，基本全是國外的IP。
普通網(wǎng)站訪問量本來就沒什么流量，這些國外的IP無疑就是最大的禍害，如果服務(wù)器按流量計(jì)費(fèi)，這些高頻率的訪問一天消耗你大量的流量費(fèi)不說，同時造成服務(wù)器資源不足，例如帶寬跑滿，CPU干到100%，內(nèi)存干到不足，造成網(wǎng)站卡頓等。
例如我在巡查我的一臺服務(wù)器時發(fā)現(xiàn)好多國外的爬蟲在抓取我的一個網(wǎng)站。
| 爬蟲類型 | 示例 IP | 所屬公司 | 地理位置 |
|---|---|---|---|
| AhrefsBot | 5.39.109.172 | Ahrefs (SEO 工具) | 法國，OVH 數(shù)據(jù)中心 |
| 51.75.236.150 | Ahrefs | 法國，OVH 數(shù)據(jù)中心 | |
| 51.75.236.141 | Ahrefs | 法國，OVH SAS，巴黎附近 | |
| SemrushBot | 85.208.96.203 | Semrush | 美國，Virginia，Ashburn |
| 185.191.171.18 | Semrush | 美國，Virginia，Ashburn 或 Texas，Dallas | |
| 185.191.171.5 | Semrush | 美國，Texas，Dallas | |
| 185.191.171.10 | Semrush | 英國，倫敦 | |
| 185.191.171.12 | Semrush | 美國，Virginia，Ashburn | |
| 185.191.171.16 | Semrush | 英國，倫敦 | |
| 85.208.96.200 | Semrush | 美國，Virginia，Ashburn | |
| DotBot (Moz) | 216.244.66.228 | Moz (SEO 工具) | 美國，Washington 州，西雅圖/圖克維拉 |
| Amazonbot | 54.235.172.108 | Amazon | 美國，Virginia，Ashburn (AWS 數(shù)據(jù)中心) |
| 44.205.192.249 | Amazon | 美國，Virginia，Ashburn (AWS) | |
| 52.200.251.20 | Amazon | 美國，Virginia，Ashburn (AWS) | |
| Applebot | 17.246.15.82 | Apple | 美國，Apple 網(wǎng)絡(luò)，屬于 Applebot |
| 17.246.23.138 | Apple | 美國，Apple 網(wǎng)絡(luò) | |
| 17.246.23.16 | Apple | 美國，加州 Cupertino（Apple 總部所在地） |
所以最好把這些IP禁止掉。要不然你的服務(wù)器受不了。
5.39.109.172 51.75.236.150 51.75.236.141 85.208.96.203 185.191.171.18 185.191.171.5 185.191.171.10 185.191.171.12 185.191.171.16 85.208.96.200 216.244.66.228 54.235.172.108 44.205.192.249 52.200.251.20 17.246.15.82 17.246.23.138 17.246.23.16 52.167.144.208 40.77.167.0/24 52.3.102.51 54.145.82.217 185.191.171.7 3.230.224.6 3.232.102.111 52.22.64.232 44.212.232.231 98.82.66.172 34.236.185.101 47.89.178.159 3.221.156.96 52.45.77.169 23.23.99.55 47.236.226.206 85.208.96.195 47.236.254.89 18.207.89.138 18.206.47.187 17.241.227.159 3.230.224.6 52.70.123.241 34.197.28.78 17.22.253.118 35.169.102.85 23.22.105.143 54.225.148.123 35.171.117.160 52.45.15.233 184.73.239.35 185.191.171.1 54.166.104.83 185.191.171.2 54.84.161.62 34.236.185.101 54.159.18.27 52.44.174.136 107.20.25.33 18.214.138.148 17.22.253.118 35.169.102.85 23.22.105.143 54.147.182.90 85.208.96.197 185.191.171.2 44.208.193.63 85.208.96.206 43.135.182.43 98.84.70.201 54.152.163.42 185.191.171.7 184.73.35.182 18.233.24.238 3.213.85.234 52.71.216.196 184.72.95.195 185.191.171.11 54.210.155.69 18.233.24.238 85.208.96.208 100.25.120.246 52.200.54.136 54.84.93.8 185.191.171.6 18.232.36.1 3.221.244.28 216.244.66.231 34.226.89.140 44.218.6.93
同時還包括其他一些攻擊和IP，我整理了一部分：
68.218.0.0/16 43.241.47.0/24 58.218.211.0/24 4.217.235.0/24 4.235.98.0/24 58.218.0.0/16 45.135.0.0/16 117.132.188.0/24 14.128.63.0/24 59.82.135.0/24 4.197.196.0/24 27.124.12.0/24 183.131.196.0/24 4.193.193.0/24 4.193.0.0/16 14.128.0.0/16 221.229.0.0/16 180.97.0.0/16 221.229.0.0/16 180.97.0.0/16 51.89.129.47 198.244.240.166 38.34.0.0/16 111.68.0.0/16 4.197.0.0/16 117.43.0.0/16 82.97.0.0/16 134.122.0.0/16 4.241.0.0/16 4.189.0.0/16 4.190.0.0/16 110.249.201.0/24 5.39.109.172 51.75.236.150 51.75.236.141 85.208.96.203 185.191.171.18 185.191.171.5 185.191.171.10 185.191.171.12 185.191.171.16 85.208.96.200 216.244.66.228 54.235.172.108 44.205.192.249 52.200.251.20 17.246.15.82 17.246.23.138 17.246.23.16 52.167.144.208 40.77.167.0/24 52.3.102.51 54.145.82.217 185.191.171.7 3.230.224.6 3.232.102.111 52.22.64.232 44.212.232.231 98.82.66.172 34.236.185.101 47.89.178.159 3.221.156.96 52.45.77.169 23.23.99.55 47.236.226.206 85.208.96.195 47.236.254.89 18.207.89.138 18.206.47.187 17.241.227.159 3.230.224.6 52.70.123.241 34.197.28.78 17.22.253.118 35.169.102.85 23.22.105.143 54.225.148.123 35.171.117.160 52.45.15.233 184.73.239.35 185.191.171.1 54.166.104.83 185.191.171.2 54.84.161.62 34.236.185.101 54.159.18.27 52.44.174.136 107.20.25.33 18.214.138.148 17.22.253.118 35.169.102.85 23.22.105.143 54.147.182.90 85.208.96.197 185.191.171.2 44.208.193.63 85.208.96.206 43.135.182.43 98.84.70.201 54.152.163.42 185.191.171.7 184.73.35.182 18.233.24.238 3.213.85.234 52.71.216.196 184.72.95.195 185.191.171.11 54.210.155.69 18.233.24.238 85.208.96.208 100.25.120.246 52.200.54.136 54.84.93.8 185.191.171.6 18.232.36.1 3.221.244.28 216.244.66.231 34.226.89.140 44.218.6.93
