Changes On Branch 158143b2224ff784

Changes In Branch remove-c11-atomics Through [158143b222] Excluding Merge-Ins

This is equivalent to a diff from 879cdc86ce to 158143b222

2020-03-31
14:30
Integrated remove-c11-atomics changes check-in: 7ee2e833d2 user: rkeene tags: trunk
14:20
Log the connect time, which is significant for closed sockets since they have no request time (since no request was made) check-in: 05f8958cc7 user: rkeene tags: remove-c11-atomics
14:19
Use a mutex (for now?) for closing idle sockets check-in: 158143b222 user: rkeene tags: remove-c11-atomics
13:41
Merged in trunk check-in: 1c1d95a764 user: rkeene tags: remove-c11-atomics
2018-05-03
20:08
Added support for not redirecting to index.html check-in: 879cdc86ce user: rkeene tags: trunk
2016-09-22
19:30
Post-release version increment check-in: f3418e68d2 user: rkeene tags: trunk

Modified Makefile from [2fd19df44e] to [8ce95f72a7].

1
2

3
4
5
6
7
8
9
1

2
3
4
5
6
7
8
9

-
+







CC = gcc
CFLAGS = -I. -std=gnu11 -Wall -W -pthread -O3 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE $(FILED_EXTRA_CFLAGS)
CFLAGS = -I. -Wall -W -pthread -O3 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE $(FILED_EXTRA_CFLAGS)
LDFLAGS = -pthread $(FILED_EXTRA_LDFLAGS)
LIBS = -lpthread $(FILED_EXTRA_LIBS)
MIMETYPES = /etc/httpd/mime.types

PREFIX = /usr/local
prefix = $(PREFIX)
bindir = $(prefix)/bin

Modified README from [e85e320923] to [dc3dfdd9b3].

51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

67
68
69

70
71
72
73
74
75
76

77
78
79
80
81
82
83
84
85

86
87
88
89
90

91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
51
52
53
54
55
56
57









58
59
60

61
62
63
64
65
66
67

68
69
70
71
72
73
74
75
76

77
78
79
80
81

82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102






















-
-
-
-
-
-
-
-
-
+


-
+






-
+








-
+




-
+




















-
-
-
-
-
-
-
-
-
-
-
-
-
-
-

   1. Logging (CFLAGS, -DFILED_DONT_LOG=1)
	It is possible to disable ALL logging from filed.  When logging is
	completely disabled interlocks (mutexes) for the logging pointer are
	not engaged and the logging functions are not compiled at all.
	This results in a slightly smaller and faster binary

   2. Kill idle connections (CFLAGS, -DFILED_DONT_TIMEOUT=1)
        Killing idle connections relies heavily upon C11 atomics.  This
        requires a relatively new version of GCC (4.9+) or other C compiler
        that implements this aspect of C11 and so it can be disabled at
        compile time (which is the only time it makes sense).  One day an
        alternate implementation might be present that uses a mutex instead
        of atomics at which point this documentation will be updated.

   3. Debugging (CFLAGS, -DFILED_DEBUG=1)
   2. Debugging (CFLAGS, -DFILED_DEBUG=1)
	This is an internal option and should only be used during development.

   4. Differing HTTP semantics (CFLAGS, -DFILED_NONBLOCK_HTTP=1)
   3. Differing HTTP semantics (CFLAGS, -DFILED_NONBLOCK_HTTP=1)
	It is possible that some HTTP clients may not process the HTTP stream
	being delivered if they cannot write to the HTTP stream itself.  This
	has not been observed yet, but it is possible.  If these semantics are
	needed (and they should not be) then they can be enabled with this
	flag at the cost of performance.

   5. Differing chroot() semantics (CFLAGS, -DFILED_FAKE_CHROOT=1)
   4. Differing chroot() semantics (CFLAGS, -DFILED_FAKE_CHROOT=1)
        In some cases it is desirable to mangle paths with a path prefix
        rather than call chroot() at startup.  This is less secure and slower
        and should be generally avoided -- however it may be necessary to do.
        In these cases the executable may be compiled with the
        FILED_FAKE_CHROOT C preprocessor macro defined and instead of calling
        chroot() all HTTP requests will have the root suffix specified as the
        argument to the "-r" or "--root" option prepended to them.

   6. Differing "index.html" handling (CFLAGS, -DFILED_DONT_REDIRECT_DIRECTORIES=1)
   5. Differing "index.html" handling (CFLAGS, -DFILED_DONT_REDIRECT_DIRECTORIES=1)
        Normally "filed" redirects users who request a directory to the
        index.html file in that directory so that no memory allocations are
        required;  This option lets the server generate the new path.

   7. MIME Types (MIMETYPES)
   6. MIME Types (MIMETYPES)
	For single-file convenience "filed" compiles the mapping of file
	extensions (the string in the filename following its last dot ("."))
	into the executable.  This mapping comes from a file in the format of
		type1   type1_extension1 type1_extension2...
		type2   type2_extension1 type2_extension2...
		...
	However it may not be desirable to include this mapping, or it may be
	desirable to use your own mapping rather than the default one.  This
	can be done by specifying the MIMETYPES macro to "make".  If no
	mapping is desired, "/dev/null" may be specified.

Log Files
---------
Because "filed" relies on chroot(2) and setuid(2), log files cannot reliably
be re-opened.  If you need log rotation then a second process, which can close
and re-open log files, must be used.  Any process may be used for writing logs
but if the process does not support log rotation then it will not provide that
benefit.  For example, if you wish to write logs to syslogd(8) you can use
logger(1), such as:
	# ./filed --root /www --user nobody --log '|logger -t filed' --daemon

Troubleshooting
---------------
   1. It won't compile, something about stdatomic.h not found or _Atomic not
      a valid type.

      => This is a bug in your compiler:
            https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58016

         GCC 4.7.x and 4.8.x define the macro indicating that they have C11
         support and do not define the macro that C11 requires to indicate
         that C11 atomics are not available.  They should define that macro.

         You can disable the features in "filed" that require C11 atomics by
         defining FILED_DONT_TIMEOUT in the Makefile.

Modified filed.c from [b6a9efc107] to [a328ae775d].

39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
39
40
41
42
43
44
45

























46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

63
64
65
66
67
68
69







-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-

















-







#include <stdarg.h>
#include <fcntl.h>
#include <stdio.h>
#include <errno.h>
#include <time.h>
#include <pwd.h>

/*
 * Determine if the C compiler supports C11 atomics
 */
#if __STDC_VERSION__ >= 201112L
#  ifndef __STDC_NO_ATOMICS__
#    define FILED_FEATURE_C11_ATOMICS 1
#  endif
#endif

/*
 * If the C compiler does not support C11 atomics, disable TIMEOUT support
 * since it relies upon it
 */
#ifndef FILED_FEATURE_C11_ATOMICS
#  define FILED_DONT_TIMEOUT 1
#endif

/*
 * These headers are only required for TIMEOUT support
 */
#ifndef FILED_DONT_TIMEOUT
#include <stdatomic.h>
#include <stdbool.h>
#endif

/* Compile time constants */
#define FILED_VERSION "1.21"
#define FILED_SENDFILE_MAX 16777215
#define FILED_MAX_FAILURE_COUNT 30
#define FILED_DEFAULT_TYPE "application/octet-stream"
#define FILED_PATH_BUFFER_SIZE 1010

/* Default values */
#define PORT 80
#define THREAD_COUNT 5
#define BIND_ADDR "::"
#define CACHE_SIZE 8209
#define LOG_FILE "-"

/* Fuzzing Test Code */
#ifdef FILED_TEST_AFL
#define FILED_DONT_LOG 1
#define FILED_DONT_TIMEOUT 1
#define pthread_create(a, x, y, z) afl_pthread_create(a, x, y, z)
#define bind(x, y, z) afl_bind(x, y, z)
#define socket(x, y, z) 8193
#define listen(x, y) 0
#define accept(x, y, z) afl_accept(x, y, z)
#define close(x) { if (strcmp(#x, "random_fd") == 0) { close(x); } else { exit(0); } }
#define fclose(x) exit(0)
633
634
635
636
637
638
639
640

641
642

643
644
645
646
647







648
649

650
651
652
653
654
655
656
657
658
659
660
661
662
663

664
665




666

667
668
669
670





671
672
673
674
675
676
677
678
679


680

681
682

683
684



685
686
687
688
689
690
691
692
693
694

695
696
697
698
699
700
701
702
703
704

705
706
707
708
709

710
711
712
713




714





715
716
717
718
719
720
721
722
723
724
725

726
727
728
729
730
731
732


733
734
735
736
737







738
739

740
741

742
743
744
745

746
747

748
749
750
751
752
753

754
755
756
757
758


759
760
761
762
763
764
765
607
608
609
610
611
612
613

614
615

616
617




618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640

641
642
643
644
645
646
647

648
649
650
651

652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667

668
669

670
671

672
673
674
675
676
677
678
679
680
681
682
683

684
685
686
687
688
689
690
691
692
693

694
695
696
697
698

699
700
701
702
703
704
705
706
707

708
709
710
711
712
713
714
715
716
717
718
719
720
721
722

723
724
725
726
727
728
729
730
731
732
733
734



735
736
737
738
739
740
741
742

743
744

745
746
747
748

749
750

751
752
753
754
755
756

757
758
759
760
761
762
763
764
765
766
767
768
769
770
771







-
+

-
+

-
-
-
-
+
+
+
+
+
+
+


+













-
+


+
+
+
+
-
+



-
+
+
+
+
+









+
+
-
+

-
+

-
+
+
+









-
+









-
+




-
+




+
+
+
+
-
+
+
+
+
+










-
+







+
+


-
-
-
+
+
+
+
+
+
+

-
+

-
+



-
+

-
+





-
+





+
+








#ifdef FILED_DONT_TIMEOUT
#define filed_sockettimeout_thread_init() 0
#define filed_sockettimeout_init() 0
#define filed_sockettimeout_accept(x) /**/
#define filed_sockettimeout_processing_start(x) /**/
#define filed_sockettimeout_processing_end(x) /**/
#define filed_sockettimeout_close(x) /**/
#define filed_sockettimeout_close(x, y) /**/
#else
_Atomic time_t filed_sockettimeout_time;
time_t filed_sockettimeout_time;
struct {
	_Atomic time_t expiration_time;
	_Atomic pthread_t thread_id;
	bool valid;
}* filed_sockettimeout_sockstatus;
	time_t expiration_time;
	pthread_t thread_id;
	enum {
		filed_sockettimeout_valid,
		filed_sockettimeout_invalid,
	} valid;
} *filed_sockettimeout_sockstatus;
long filed_sockettimeout_sockstatus_length;
int filed_sockettimeout_devnull_fd;
pthread_mutex_t filed_sockettimeout_mutex = PTHREAD_MUTEX_INITIALIZER;

static int filed_sockettimeout_sockfd_in_range(int sockfd) {
	if (sockfd < 3) {
		return(0);
	}

	if (sockfd > filed_sockettimeout_sockstatus_length) {
		return(0);
	}

	return(1);
}

static void filed_sockettimeout_expire(int sockfd, int length) {
static void filed_sockettimeout_expire(int sockfd, int length, int lockheld) {
	time_t now, expire;

	if (!lockheld) {
		pthread_mutex_lock(&filed_sockettimeout_mutex);
	}

	now = atomic_load(&filed_sockettimeout_time);
	now = filed_sockettimeout_time;

	expire = now + length;

	atomic_store(&filed_sockettimeout_sockstatus[sockfd].expiration_time, expire);
	filed_sockettimeout_sockstatus[sockfd].expiration_time = expire;

	if (!lockheld) {
		pthread_mutex_unlock(&filed_sockettimeout_mutex);
	}

	return;
}

static void filed_sockettimeout_accept(int sockfd) {
	if (!filed_sockettimeout_sockfd_in_range(sockfd)) {
		return;
	}

	pthread_mutex_lock(&filed_sockettimeout_mutex);

	filed_sockettimeout_expire(sockfd, 60);
	filed_sockettimeout_expire(sockfd, 60, 1);

	atomic_store(&filed_sockettimeout_sockstatus[sockfd].thread_id, pthread_self());
	filed_sockettimeout_sockstatus[sockfd].thread_id = pthread_self();

	atomic_store(&filed_sockettimeout_sockstatus[sockfd].valid, true);
	filed_sockettimeout_sockstatus[sockfd].valid = filed_sockettimeout_valid;

	pthread_mutex_unlock(&filed_sockettimeout_mutex);

	return;
}

static void filed_sockettimeout_processing_start(int sockfd) {
	if (!filed_sockettimeout_sockfd_in_range(sockfd)) {
		return;
	}

	filed_sockettimeout_expire(sockfd, 86400);
	filed_sockettimeout_expire(sockfd, 86400, 0);

	return;
}

static void filed_sockettimeout_processing_end(int sockfd) {
	if (!filed_sockettimeout_sockfd_in_range(sockfd)) {
		return;
	}

	filed_sockettimeout_expire(sockfd, 60);
	filed_sockettimeout_expire(sockfd, 60, 0);

	return;
}

static void filed_sockettimeout_close(int sockfd) {
static void filed_sockettimeout_close(int sockfd, int lockheld) {
	if (!filed_sockettimeout_sockfd_in_range(sockfd)) {
		return;
	}

	if (!lockheld) {
		pthread_mutex_lock(&filed_sockettimeout_mutex);
	}

	atomic_store(&filed_sockettimeout_sockstatus[sockfd].valid, false);
	filed_sockettimeout_sockstatus[sockfd].valid = filed_sockettimeout_invalid;

	if (!lockheld) {
		pthread_mutex_unlock(&filed_sockettimeout_mutex);
	}

	return;
}

static void *filed_sockettimeout_thread(void *arg) {
	struct timespec sleep_time;
	time_t now, expiration_time;
	pthread_t thread_id;
	long idx;
	int count;
	bool valid;
	int valid;

	while (1) {
		for (count = 0; count < 10; count++) {
			sleep_time.tv_sec = 30;
			sleep_time.tv_nsec = 0;
			nanosleep(&sleep_time, NULL);

			pthread_mutex_lock(&filed_sockettimeout_mutex);

			now = time(NULL);

			atomic_store(&filed_sockettimeout_time, now);
		}

			filed_sockettimeout_time = now;

			pthread_mutex_unlock(&filed_sockettimeout_mutex);
		}

		pthread_mutex_lock(&filed_sockettimeout_mutex);

		for (idx = 0; idx < filed_sockettimeout_sockstatus_length; idx++) {
			valid = atomic_load(&filed_sockettimeout_sockstatus[idx].valid);
			valid = filed_sockettimeout_sockstatus[idx].valid;

			if (!valid) {
			if (valid != filed_sockettimeout_valid) {
				continue;
			}

			expiration_time = atomic_load(&filed_sockettimeout_sockstatus[idx].expiration_time);
			expiration_time = filed_sockettimeout_sockstatus[idx].expiration_time;

			thread_id = atomic_load(&filed_sockettimeout_sockstatus[idx].thread_id);
			thread_id = filed_sockettimeout_sockstatus[idx].thread_id;

			if (expiration_time > now) {
				continue;
			}

			filed_sockettimeout_close(idx);
			filed_sockettimeout_close(idx, 1);

			dup2(filed_sockettimeout_devnull_fd, idx);

			pthread_kill(thread_id, SIGPIPE);
		}

		pthread_mutex_unlock(&filed_sockettimeout_mutex);
	}

	return(NULL);

	/* NOTREACH: We don't actually take any arguments */
	arg = arg;
}
776
777
778
779
780
781
782

783

784
785
786
787
788
789

790
791
792
793
794
795
796
797
798
799
782
783
784
785
786
787
788
789

790
791
792
793
794
795

796
797
798

799
800
801
802
803
804
805







+
-
+





-
+


-







	long maxfd, idx;

	maxfd = sysconf(_SC_OPEN_MAX);
	if (maxfd <= 0) {
		maxfd = 4096;
	}

	filed_sockettimeout_sockstatus_length = maxfd;
	filed_sockettimeout_sockstatus = malloc(sizeof(*filed_sockettimeout_sockstatus) * maxfd);
	filed_sockettimeout_sockstatus = malloc(sizeof(*filed_sockettimeout_sockstatus) * filed_sockettimeout_sockstatus_length);
	if (filed_sockettimeout_sockstatus == NULL) {
		return(-1);
	}

	for (idx = 0; idx < maxfd; idx++) {
		filed_sockettimeout_sockstatus[idx].valid = false;
		filed_sockettimeout_sockstatus[idx].valid = filed_sockettimeout_invalid;
	}

	filed_sockettimeout_sockstatus_length = maxfd;
	filed_sockettimeout_devnull_fd = open("/dev/null", O_RDWR);
	if (filed_sockettimeout_devnull_fd < 0) {
		return(-1);
	}

	return(0);
}
1199
1200
1201
1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212
1213
1205
1206
1207
1208
1209
1210
1211

1212
1213
1214
1215
1216
1217
1218
1219







-
+







	/** reason must point to a globally allocated value **/
	log->reason = reason;
	log->http_code = error_number;

	filed_log_entry(log);

	/* Close connection */
	filed_sockettimeout_close(fileno(fp));
	filed_sockettimeout_close(fileno(fp), 0);

	fclose(fp);

	return;
}

/* Return a redirect to index.html */
1224
1225
1226
1227
1228
1229
1230
1231

1232
1233
1234
1235
1236
1237
1238
1230
1231
1232
1233
1234
1235
1236

1237
1238
1239
1240
1241
1242
1243
1244







-
+







	/* Log redirect */
	log->reason = "redirect";
	log->http_code = http_code;

	filed_log_entry(log);

	/* Close connection */
	filed_sockettimeout_close(fileno(fp));
	filed_sockettimeout_close(fileno(fp), 0);

	fclose(fp);

	return;

	/* Currently unused: path */
	path = path;
1264
1265
1266
1267
1268
1269
1270
1271

1272
1273
1274
1275
1276
1277
1278
1270
1271
1272
1273
1274
1275
1276

1277
1278
1279
1280
1281
1282
1283
1284







-
+








	/* Determine current time */
	date_current = filed_format_time(date_current_b, sizeof(date_current_b), time(NULL));

	/* Open socket as ANSI I/O for ease of use */
	fp = fdopen(fd, "w+b");
	if (fp == NULL) {
		filed_sockettimeout_close(fd);
		filed_sockettimeout_close(fd, 0);

		close(fd);

		log->buffer[0] = '\0';
		log->http_code = -1;
		log->reason = "fdopen_failed";

1457
1458
1459
1460
1461
1462
1463
1464

1465
1466
1467
1468
1469
1470
1471
1463
1464
1465
1466
1467
1468
1469

1470
1471
1472
1473
1474
1475
1476
1477







-
+







	log->sent_length = sendfile_sent;

	filed_log_entry(log);

	close(fileinfo->fd);

	if (request->headers.connection != FILED_CONNECTION_KEEP_ALIVE) {
		filed_sockettimeout_close(fd);
		filed_sockettimeout_close(fd, 0);

		fclose(fp);

		return(FILED_CONNECTION_CLOSE);
	}

	filed_sockettimeout_processing_end(fd);