Overview
| Comment: | Integrated remove-c11-atomics changes |
|---|---|
| Downloads: | Tarball | ZIP archive | SQL archive |
| Timelines: | family | ancestors | descendants | both | trunk |
| Files: | files | file ages | folders |
| SHA1: | 7ee2e833d2dd190ae37fa965e8d124805f701bea |
| User & Date: | rkeene on 2020-03-31 14:30:06 |
| Other Links: | manifest | tags |
Context
|
2020-03-31
| ||
| 18:46 | Added AFL test script check-in: 05a7ef3fae user: rkeene tags: trunk | |
| 14:33 | Merged in trunk check-in: 2204669e3b user: rkeene tags: seccomp | |
| 14:30 | Integrated remove-c11-atomics changes check-in: 7ee2e833d2 user: rkeene tags: trunk | |
| 14:26 | Made socket idle timeout checks more frequent Closed-Leaf check-in: fe6d401e68 user: rkeene tags: remove-c11-atomics | |
|
2018-05-03
| ||
| 20:08 | Added support for not redirecting to index.html check-in: 879cdc86ce user: rkeene tags: trunk | |
Changes
Modified Makefile from [2fd19df44e] to [8ce95f72a7].
1 1 CC = gcc 2 -CFLAGS = -I. -std=gnu11 -Wall -W -pthread -O3 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE $(FILED_EXTRA_CFLAGS) 2 +CFLAGS = -I. -Wall -W -pthread -O3 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE $(FILED_EXTRA_CFLAGS) 3 3 LDFLAGS = -pthread $(FILED_EXTRA_LDFLAGS) 4 4 LIBS = -lpthread $(FILED_EXTRA_LIBS) 5 5 MIMETYPES = /etc/httpd/mime.types 6 6 7 7 PREFIX = /usr/local 8 8 prefix = $(PREFIX) 9 9 bindir = $(prefix)/bin
Modified README from [e85e320923] to [dc3dfdd9b3].
51 51 52 52 1. Logging (CFLAGS, -DFILED_DONT_LOG=1) 53 53 It is possible to disable ALL logging from filed. When logging is 54 54 completely disabled interlocks (mutexes) for the logging pointer are 55 55 not engaged and the logging functions are not compiled at all. 56 56 This results in a slightly smaller and faster binary 57 57 58 - 2. Kill idle connections (CFLAGS, -DFILED_DONT_TIMEOUT=1) 59 - Killing idle connections relies heavily upon C11 atomics. This 60 - requires a relatively new version of GCC (4.9+) or other C compiler 61 - that implements this aspect of C11 and so it can be disabled at 62 - compile time (which is the only time it makes sense). One day an 63 - alternate implementation might be present that uses a mutex instead 64 - of atomics at which point this documentation will be updated. 65 - 66 - 3. Debugging (CFLAGS, -DFILED_DEBUG=1) 58 + 2. Debugging (CFLAGS, -DFILED_DEBUG=1) 67 59 This is an internal option and should only be used during development. 68 60 69 - 4. Differing HTTP semantics (CFLAGS, -DFILED_NONBLOCK_HTTP=1) 61 + 3. Differing HTTP semantics (CFLAGS, -DFILED_NONBLOCK_HTTP=1) 70 62 It is possible that some HTTP clients may not process the HTTP stream 71 63 being delivered if they cannot write to the HTTP stream itself. This 72 64 has not been observed yet, but it is possible. If these semantics are 73 65 needed (and they should not be) then they can be enabled with this 74 66 flag at the cost of performance. 75 67 76 - 5. Differing chroot() semantics (CFLAGS, -DFILED_FAKE_CHROOT=1) 68 + 4. Differing chroot() semantics (CFLAGS, -DFILED_FAKE_CHROOT=1) 77 69 In some cases it is desirable to mangle paths with a path prefix 78 70 rather than call chroot() at startup. This is less secure and slower 79 71 and should be generally avoided -- however it may be necessary to do. 80 72 In these cases the executable may be compiled with the 81 73 FILED_FAKE_CHROOT C preprocessor macro defined and instead of calling 82 74 chroot() all HTTP requests will have the root suffix specified as the 83 75 argument to the "-r" or "--root" option prepended to them. 84 76 85 - 6. Differing "index.html" handling (CFLAGS, -DFILED_DONT_REDIRECT_DIRECTORIES=1) 77 + 5. Differing "index.html" handling (CFLAGS, -DFILED_DONT_REDIRECT_DIRECTORIES=1) 86 78 Normally "filed" redirects users who request a directory to the 87 79 index.html file in that directory so that no memory allocations are 88 80 required; This option lets the server generate the new path. 89 81 90 - 7. MIME Types (MIMETYPES) 82 + 6. MIME Types (MIMETYPES) 91 83 For single-file convenience "filed" compiles the mapping of file 92 84 extensions (the string in the filename following its last dot (".")) 93 85 into the executable. This mapping comes from a file in the format of 94 86 type1 type1_extension1 type1_extension2... 95 87 type2 type2_extension1 type2_extension2... 96 88 ... 97 89 However it may not be desirable to include this mapping, or it may be ................................................................................ 104 96 Because "filed" relies on chroot(2) and setuid(2), log files cannot reliably 105 97 be re-opened. If you need log rotation then a second process, which can close 106 98 and re-open log files, must be used. Any process may be used for writing logs 107 99 but if the process does not support log rotation then it will not provide that 108 100 benefit. For example, if you wish to write logs to syslogd(8) you can use 109 101 logger(1), such as: 110 102 # ./filed --root /www --user nobody --log '|logger -t filed' --daemon 111 - 112 -Troubleshooting 113 ---------------- 114 - 1. It won't compile, something about stdatomic.h not found or _Atomic not 115 - a valid type. 116 - 117 - => This is a bug in your compiler: 118 - https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58016 119 - 120 - GCC 4.7.x and 4.8.x define the macro indicating that they have C11 121 - support and do not define the macro that C11 requires to indicate 122 - that C11 atomics are not available. They should define that macro. 123 - 124 - You can disable the features in "filed" that require C11 atomics by 125 - defining FILED_DONT_TIMEOUT in the Makefile.
Modified filed.c from [b6a9efc107] to [7252b16239].
39 39 #include <stdarg.h> 40 40 #include <fcntl.h> 41 41 #include <stdio.h> 42 42 #include <errno.h> 43 43 #include <time.h> 44 44 #include <pwd.h> 45 45 46 -/* 47 - * Determine if the C compiler supports C11 atomics 48 - */ 49 -#if __STDC_VERSION__ >= 201112L 50 -# ifndef __STDC_NO_ATOMICS__ 51 -# define FILED_FEATURE_C11_ATOMICS 1 52 -# endif 53 -#endif 54 - 55 -/* 56 - * If the C compiler does not support C11 atomics, disable TIMEOUT support 57 - * since it relies upon it 58 - */ 59 -#ifndef FILED_FEATURE_C11_ATOMICS 60 -# define FILED_DONT_TIMEOUT 1 61 -#endif 62 - 63 -/* 64 - * These headers are only required for TIMEOUT support 65 - */ 66 -#ifndef FILED_DONT_TIMEOUT 67 -#include <stdatomic.h> 68 -#include <stdbool.h> 69 -#endif 70 - 71 46 /* Compile time constants */ 72 47 #define FILED_VERSION "1.21" 73 48 #define FILED_SENDFILE_MAX 16777215 74 49 #define FILED_MAX_FAILURE_COUNT 30 75 50 #define FILED_DEFAULT_TYPE "application/octet-stream" 76 51 #define FILED_PATH_BUFFER_SIZE 1010 77 52 ................................................................................ 81 56 #define BIND_ADDR "::" 82 57 #define CACHE_SIZE 8209 83 58 #define LOG_FILE "-" 84 59 85 60 /* Fuzzing Test Code */ 86 61 #ifdef FILED_TEST_AFL 87 62 #define FILED_DONT_LOG 1 88 -#define FILED_DONT_TIMEOUT 1 89 63 #define pthread_create(a, x, y, z) afl_pthread_create(a, x, y, z) 90 64 #define bind(x, y, z) afl_bind(x, y, z) 91 65 #define socket(x, y, z) 8193 92 66 #define listen(x, y) 0 93 67 #define accept(x, y, z) afl_accept(x, y, z) 94 68 #define close(x) { if (strcmp(#x, "random_fd") == 0) { close(x); } else { exit(0); } } 95 69 #define fclose(x) exit(0) ................................................................................ 204 178 /* Message buffer for type = MESSAGE */ 205 179 /* Path buffer for type = TRANSFER */ 206 180 char buffer[FILED_PATH_BUFFER_SIZE]; 207 181 208 182 /* Items for type = TRANSFER */ 209 183 int http_code; 210 184 const char *reason; 185 + time_t connecttime; 211 186 time_t starttime; 212 187 time_t endtime; 213 188 off_t req_offset; 214 189 off_t req_length; 215 190 off_t sent_length; 216 191 off_t file_length; 217 192 char ip[128]; ................................................................................ 486 461 break; 487 462 } 488 463 489 464 if (curr->endtime == ((time_t) -1)) { 490 465 curr->endtime = now; 491 466 } 492 467 493 - fprintf(fp, "TRANSFER METHOD=%s PATH=%s SRC=%s:%i TIME.START=%llu TIME.END=%llu CODE.VALUE=%u CODE.REASON=%s REQUEST.OFFSET=%llu REQUEST.LENGTH=%llu FILE.LENGTH=%llu TRANSFER.LENGTH=%llu", 468 + fprintf(fp, "TRANSFER METHOD=%s PATH=%s SRC=%s:%i CLIENT.TIME.CONNECT=%llu REQUEST.TIME.START=%llu REQUEST.TIME.END=%llu CODE.VALUE=%u CODE.REASON=%s REQUEST.OFFSET=%llu REQUEST.LENGTH=%llu FILE.LENGTH=%llu TRANSFER.LENGTH=%llu", 494 469 method, 495 470 curr->buffer, 496 471 curr->ip, curr->port, 472 + (unsigned long long) curr->connecttime, 497 473 (unsigned long long) curr->starttime, 498 474 (unsigned long long) curr->endtime, 499 475 curr->http_code, curr->reason, 500 476 (unsigned long long) curr->req_offset, 501 477 (unsigned long long) curr->req_length, 502 478 (unsigned long long) curr->file_length, 503 479 (unsigned long long) curr->sent_length ................................................................................ 540 516 struct filed_log_entry *retval; 541 517 542 518 retval = malloc(sizeof(*retval)); 543 519 544 520 if (initialize) { 545 521 retval->buffer[0] = '\0'; 546 522 retval->http_code = -1; 523 + retval->connecttime = 0; 547 524 retval->starttime = 0; 548 525 retval->endtime = 0; 549 526 retval->req_offset = 0; 550 527 retval->req_length = 0; 551 528 retval->sent_length = 0; 552 529 retval->file_length = 0; 553 530 retval->ip[0] = '\0'; ................................................................................ 633 610 634 611 #ifdef FILED_DONT_TIMEOUT 635 612 #define filed_sockettimeout_thread_init() 0 636 613 #define filed_sockettimeout_init() 0 637 614 #define filed_sockettimeout_accept(x) /**/ 638 615 #define filed_sockettimeout_processing_start(x) /**/ 639 616 #define filed_sockettimeout_processing_end(x) /**/ 640 -#define filed_sockettimeout_close(x) /**/ 617 +#define filed_sockettimeout_close(x, y) /**/ 641 618 #else 642 -_Atomic time_t filed_sockettimeout_time; 619 +time_t filed_sockettimeout_time; 643 620 struct { 644 - _Atomic time_t expiration_time; 645 - _Atomic pthread_t thread_id; 646 - bool valid; 647 -}* filed_sockettimeout_sockstatus; 621 + time_t expiration_time; 622 + pthread_t thread_id; 623 + enum { 624 + filed_sockettimeout_valid, 625 + filed_sockettimeout_invalid, 626 + } valid; 627 +} *filed_sockettimeout_sockstatus; 648 628 long filed_sockettimeout_sockstatus_length; 649 629 int filed_sockettimeout_devnull_fd; 630 +pthread_mutex_t filed_sockettimeout_mutex = PTHREAD_MUTEX_INITIALIZER; 650 631 651 632 static int filed_sockettimeout_sockfd_in_range(int sockfd) { 652 633 if (sockfd < 3) { 653 634 return(0); 654 635 } 655 636 656 637 if (sockfd > filed_sockettimeout_sockstatus_length) { 657 638 return(0); 658 639 } 659 640 660 641 return(1); 661 642 } 662 643 663 -static void filed_sockettimeout_expire(int sockfd, int length) { 644 +static void filed_sockettimeout_expire(int sockfd, int length, int lockheld) { 664 645 time_t now, expire; 665 646 666 - now = atomic_load(&filed_sockettimeout_time); 647 + if (!lockheld) { 648 + pthread_mutex_lock(&filed_sockettimeout_mutex); 649 + } 650 + 651 + now = filed_sockettimeout_time; 667 652 668 653 expire = now + length; 669 654 670 - atomic_store(&filed_sockettimeout_sockstatus[sockfd].expiration_time, expire); 655 + filed_sockettimeout_sockstatus[sockfd].expiration_time = expire; 656 + 657 + if (!lockheld) { 658 + pthread_mutex_unlock(&filed_sockettimeout_mutex); 659 + } 671 660 672 661 return; 673 662 } 674 663 675 664 static void filed_sockettimeout_accept(int sockfd) { 676 665 if (!filed_sockettimeout_sockfd_in_range(sockfd)) { 677 666 return; 678 667 } 679 668 680 - filed_sockettimeout_expire(sockfd, 60); 669 + pthread_mutex_lock(&filed_sockettimeout_mutex); 681 670 682 - atomic_store(&filed_sockettimeout_sockstatus[sockfd].thread_id, pthread_self()); 671 + filed_sockettimeout_expire(sockfd, 60, 1); 683 672 684 - atomic_store(&filed_sockettimeout_sockstatus[sockfd].valid, true); 673 + filed_sockettimeout_sockstatus[sockfd].thread_id = pthread_self(); 674 + 675 + filed_sockettimeout_sockstatus[sockfd].valid = filed_sockettimeout_valid; 676 + 677 + pthread_mutex_unlock(&filed_sockettimeout_mutex); 685 678 686 679 return; 687 680 } 688 681 689 682 static void filed_sockettimeout_processing_start(int sockfd) { 690 683 if (!filed_sockettimeout_sockfd_in_range(sockfd)) { 691 684 return; 692 685 } 693 686 694 - filed_sockettimeout_expire(sockfd, 86400); 687 + filed_sockettimeout_expire(sockfd, 86400, 0); 695 688 696 689 return; 697 690 } 698 691 699 692 static void filed_sockettimeout_processing_end(int sockfd) { 700 693 if (!filed_sockettimeout_sockfd_in_range(sockfd)) { 701 694 return; 702 695 } 703 696 704 - filed_sockettimeout_expire(sockfd, 60); 697 + filed_sockettimeout_expire(sockfd, 60, 0); 705 698 706 699 return; 707 700 } 708 701 709 -static void filed_sockettimeout_close(int sockfd) { 702 +static void filed_sockettimeout_close(int sockfd, int lockheld) { 710 703 if (!filed_sockettimeout_sockfd_in_range(sockfd)) { 711 704 return; 712 705 } 713 706 714 - atomic_store(&filed_sockettimeout_sockstatus[sockfd].valid, false); 707 + if (!lockheld) { 708 + pthread_mutex_lock(&filed_sockettimeout_mutex); 709 + } 710 + 711 + filed_sockettimeout_sockstatus[sockfd].valid = filed_sockettimeout_invalid; 712 + 713 + if (!lockheld) { 714 + pthread_mutex_unlock(&filed_sockettimeout_mutex); 715 + } 715 716 716 717 return; 717 718 } 718 719 719 720 static void *filed_sockettimeout_thread(void *arg) { 720 721 struct timespec sleep_time; 721 722 time_t now, expiration_time; 722 723 pthread_t thread_id; 723 724 long idx; 724 725 int count; 725 - bool valid; 726 + int valid; 727 + int time_interval = 30; 728 + int check_period = 90; 726 729 727 730 while (1) { 728 - for (count = 0; count < 10; count++) { 729 - sleep_time.tv_sec = 30; 731 + for (count = 0; count < (check_period / time_interval); count++) { 732 + sleep_time.tv_sec = time_interval; 730 733 sleep_time.tv_nsec = 0; 731 734 nanosleep(&sleep_time, NULL); 732 735 736 + pthread_mutex_lock(&filed_sockettimeout_mutex); 737 + 733 738 now = time(NULL); 734 739 735 - atomic_store(&filed_sockettimeout_time, now); 740 + filed_sockettimeout_time = now; 741 + 742 + pthread_mutex_unlock(&filed_sockettimeout_mutex); 736 743 } 744 + 745 + pthread_mutex_lock(&filed_sockettimeout_mutex); 737 746 738 747 for (idx = 0; idx < filed_sockettimeout_sockstatus_length; idx++) { 739 - valid = atomic_load(&filed_sockettimeout_sockstatus[idx].valid); 748 + valid = filed_sockettimeout_sockstatus[idx].valid; 740 749 741 - if (!valid) { 750 + if (valid != filed_sockettimeout_valid) { 742 751 continue; 743 752 } 744 753 745 - expiration_time = atomic_load(&filed_sockettimeout_sockstatus[idx].expiration_time); 754 + expiration_time = filed_sockettimeout_sockstatus[idx].expiration_time; 746 755 747 - thread_id = atomic_load(&filed_sockettimeout_sockstatus[idx].thread_id); 756 + thread_id = filed_sockettimeout_sockstatus[idx].thread_id; 748 757 749 758 if (expiration_time > now) { 750 759 continue; 751 760 } 752 761 753 - filed_sockettimeout_close(idx); 762 + filed_sockettimeout_close(idx, 1); 754 763 755 764 dup2(filed_sockettimeout_devnull_fd, idx); 756 765 757 766 pthread_kill(thread_id, SIGPIPE); 758 767 } 768 + 769 + pthread_mutex_unlock(&filed_sockettimeout_mutex); 759 770 } 760 771 761 772 return(NULL); 762 773 763 774 /* NOTREACH: We don't actually take any arguments */ 764 775 arg = arg; 765 776 } ................................................................................ 776 787 long maxfd, idx; 777 788 778 789 maxfd = sysconf(_SC_OPEN_MAX); 779 790 if (maxfd <= 0) { 780 791 maxfd = 4096; 781 792 } 782 793 783 - filed_sockettimeout_sockstatus = malloc(sizeof(*filed_sockettimeout_sockstatus) * maxfd); 794 + filed_sockettimeout_sockstatus_length = maxfd; 795 + filed_sockettimeout_sockstatus = malloc(sizeof(*filed_sockettimeout_sockstatus) * filed_sockettimeout_sockstatus_length); 784 796 if (filed_sockettimeout_sockstatus == NULL) { 785 797 return(-1); 786 798 } 787 799 788 800 for (idx = 0; idx < maxfd; idx++) { 789 - filed_sockettimeout_sockstatus[idx].valid = false; 801 + filed_sockettimeout_sockstatus[idx].valid = filed_sockettimeout_invalid; 790 802 } 791 803 792 - filed_sockettimeout_sockstatus_length = maxfd; 793 804 filed_sockettimeout_devnull_fd = open("/dev/null", O_RDWR); 794 805 if (filed_sockettimeout_devnull_fd < 0) { 795 806 return(-1); 796 807 } 797 808 798 809 return(0); 799 810 } ................................................................................ 1199 1210 /** reason must point to a globally allocated value **/ 1200 1211 log->reason = reason; 1201 1212 log->http_code = error_number; 1202 1213 1203 1214 filed_log_entry(log); 1204 1215 1205 1216 /* Close connection */ 1206 - filed_sockettimeout_close(fileno(fp)); 1217 + filed_sockettimeout_close(fileno(fp), 0); 1207 1218 1208 1219 fclose(fp); 1209 1220 1210 1221 return; 1211 1222 } 1212 1223 1213 1224 /* Return a redirect to index.html */ ................................................................................ 1224 1235 /* Log redirect */ 1225 1236 log->reason = "redirect"; 1226 1237 log->http_code = http_code; 1227 1238 1228 1239 filed_log_entry(log); 1229 1240 1230 1241 /* Close connection */ 1231 - filed_sockettimeout_close(fileno(fp)); 1242 + filed_sockettimeout_close(fileno(fp), 0); 1232 1243 1233 1244 fclose(fp); 1234 1245 1235 1246 return; 1236 1247 1237 1248 /* Currently unused: path */ 1238 1249 path = path; ................................................................................ 1258 1269 size_t sendfile_size; 1259 1270 off_t sendfile_offset, sendfile_sent, sendfile_len; 1260 1271 char *path; 1261 1272 char *date_current, date_current_b[64]; 1262 1273 int http_code; 1263 1274 FILE *fp; 1264 1275 1276 + /* Indicate the connection start time */ 1277 + log->connecttime = time(NULL); 1278 + 1265 1279 /* Determine current time */ 1266 1280 date_current = filed_format_time(date_current_b, sizeof(date_current_b), time(NULL)); 1267 1281 1268 1282 /* Open socket as ANSI I/O for ease of use */ 1269 1283 fp = fdopen(fd, "w+b"); 1270 1284 if (fp == NULL) { 1271 - filed_sockettimeout_close(fd); 1285 + filed_sockettimeout_close(fd, 0); 1272 1286 1273 1287 close(fd); 1274 1288 1275 1289 log->buffer[0] = '\0'; 1276 1290 log->http_code = -1; 1277 1291 log->reason = "fdopen_failed"; 1278 1292 ................................................................................ 1457 1471 log->sent_length = sendfile_sent; 1458 1472 1459 1473 filed_log_entry(log); 1460 1474 1461 1475 close(fileinfo->fd); 1462 1476 1463 1477 if (request->headers.connection != FILED_CONNECTION_KEEP_ALIVE) { 1464 - filed_sockettimeout_close(fd); 1478 + filed_sockettimeout_close(fd, 0); 1465 1479 1466 1480 fclose(fp); 1467 1481 1468 1482 return(FILED_CONNECTION_CLOSE); 1469 1483 } 1470 1484 1471 1485 filed_sockettimeout_processing_end(fd);